spec-eval 0.2.0__tar.gz

spec_eval-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ben Jones
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spec_eval-0.2.0/PKG-INFO

@@ -0,0 +1,293 @@
+Metadata-Version: 2.4
+Name: spec-eval
+Version: 0.2.0
+Summary: A trust-eval for your specs: coverage, drift, and sufficiency of markdown specs against code.
+Author: Ben Jones
+License: MIT
+Project-URL: Homepage, https://github.com/benjaminjones/spec-eval
+Project-URL: Repository, https://github.com/benjaminjones/spec-eval
+Project-URL: Issues, https://github.com/benjaminjones/spec-eval/issues
+Keywords: spec-driven-development,documentation-drift,docs-as-code,living-documentation,llm-as-judge,code-documentation,backfill-documentation
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Documentation
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anthropic>=0.40
+Requires-Dist: openai>=1.50
+Requires-Dist: google-genai>=1.0
+Requires-Dist: PyYAML>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: hypothesis>=6; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Dynamic: license-file
+
+# spec-eval — Build fast with AI. Validate with specs.
+
+**Your code changes fast. Your docs fall behind. spec-eval keeps them honest.**
+
+*The trust layer for [spec-driven development](https://en.wikipedia.org/wiki/Spec-driven_development) — iterate fast, keep the spec and code in agreement.*
+
+*A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.*
+
+[Get started](#try-it) · [Tutorial](GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](FAQ.md)
+
+A **spec** is short for **specification** — a plain-English description of what your code should do. **spec-eval**
+keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth
+trusting. How far you lean on it is up to you. **No specs yet? spec-eval writes them for you.**
+
+## Table of contents
+
+- [The problem](#the-problem)
+- [What it checks](#what-it-checks)
+- [What it creates](#what-it-creates)
+- [Try it](#try-it)
+  - [Run via prompt chat (no setup)](#run-via-prompt-chat-no-setup)
+  - [Run in the terminal, no key (Claude subscription)](#run-in-the-terminal-no-key-claude-subscription)
+  - [Run in the terminal with an API key (for CI)](#run-in-the-terminal-with-an-api-key-for-ci)
+- [How the scores are made](#how-the-scores-are-made)
+- [Reading the scores](#reading-the-scores)
+- [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion)
+- [Proof it works](#proof-it-works)
+- [More](#more)
+- [FAQ](#faq)
+- [Feedback](#feedback)
+- [Acknowledgements](#acknowledgements)
+
+## The problem
+
+Specs go stale the moment code moves on — and one stale spec is enough to stop a reader trusting all of them.
+AI coding agents make code move faster than any hand-maintained doc can follow.
+
+**spec-eval** treats specs like code: **run a check — via a chat prompt, an ordinary command, or a CI step — and
+the spec is measured against the source.** A spec you can check is a spec you can trust.
+
+## What it checks
+
+Three simple questions:
+
+| Check | The question it answers | How |
+|---|---|---|
+| **coverage** | Does this code have a spec at all? | Just counts files. Free — no AI, no key. |
+| **drift** | Does the spec say something the code doesn't do? | An AI reads each spec and its code and lists the clashes. |
+| **sufficiency** | How completely does the spec capture the code's behavior? | An AI lists what the spec misses and scores it 0–1 — an indicator, not a guarantee. |
+
+Your **code is the source of truth** — the spec is what gets graded against it.
+
+## What it creates
+
+Everything it writes is plain markdown, right next to your code:
+
+```text
+ your code                 specs beside it              spec-reports/
+┌────────────┐             ┌────────────┐             ┌────────────────┐
+│ parser.py  │  generate   │ parser.md  │  coverage   │ coverage.md    │
+│ api.py     │ ──────────► │ api.md     │  audit      │ report.md      │
+│ …          │             │ …          │ ──────────► │ sufficiency.md │
+└────────────┘             └────────────┘ sufficiency └───────┬────────┘
+                                                              │
+                                                              ▼
+                                                       SPEC-HEALTH.md
+                                                    (one-page scorecard)
+```
+
+## Try it
+
+Prefer a guided walkthrough? **[Getting-started tutorial →](GETTING-STARTED.md)**
+
+**The commands** — in the order you'll typically use them:
+
+- **coverage** — which files have no spec yet? *(free — no AI)*
+- **generate** — writes a plain-English spec beside each file that has none *(uses AI)*
+- **sufficiency** — how completely does each spec capture the code? Scored 0–1 *(uses AI)*
+- **audit** — the drift check: does any spec clash with the code? Run it at every milestone *(uses AI)*
+
+None of them change your files — reports go to `spec-reports/`. The exception is `generate`, which writes new
+specs beside your code: review them and commit the keepers like code.
+
+The `(uses AI)` commands (`generate`, `sufficiency`, `audit`) need an AI behind them — `coverage` never does.
+Three ways to run them, least setup first:
+
+### Run via prompt chat (no setup)
+
+Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
+land in the chat:
+```text
+Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
+first check coverage, then check my specs against my code.
+```
+The path is wherever you cloned this repo — no clone? Give your agent [the file's link](skills/spec-check/SKILL.md).
+
+Optionally save reports — add:
+```text
+and save the results to spec-reports/
+```
+The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).
+
+**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](skills/) into your agent's skills
+folder (Claude Code: `.claude/skills/`). Then the prompt is the command:
+```bash
+mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/
+```
+```text
+Which files have no spec yet?        # coverage
+Write a spec for src/parser.py       # generate
+Check my specs against my code       # sufficiency + audit
+```
+
+### Run in the terminal, no key (Claude subscription)
+
+Install once, then add `--model claude-code` to any `(uses AI)` command — **spec-eval** routes through the
+`claude` CLI you're already logged into (no key; your `ANTHROPIC_API_KEY` is hidden from the call, so it can't
+accidentally bill the paid API). The same four commands, in the same order as the list above:
+```bash
+pip install spec-eval                                     # or, from a clone:  pip install -e .
+spec-eval coverage    ./your-project                      # free — no AI, no key
+spec-eval generate    ./your-project --model claude-code
+spec-eval sufficiency ./your-project --model claude-code
+spec-eval audit       ./your-project --model claude-code
+```
+Every terminal run writes its reports to `spec-reports/` (plus a `runs.jsonl` history line).
+
+### Run in the terminal with an API key (for CI)
+
+Export a key and run any command above **without** `--model claude-code` — an API key is the default. This is
+the option for CI, the checks that run automatically on every push (store the key as a repo secret):
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...     # or OPENAI_API_KEY / GOOGLE_API_KEY
+```
+
+> [!NOTE]
+> A typical first run is `coverage → generate → sufficiency`, then at every milestone after: `audit` — plus
+> `sufficiency` when you've added behavior. The [tutorial](GETTING-STARTED.md) has step-by-step guides for
+> starting with no specs and for existing docs.
+
+> [!TIP]
+> **Version control is also your preview.** The three checks never modify your project; `generate` adds
+> ordinary new files — review them with `git diff`, edit what's off, and drop any you don't want with
+> `git checkout -- <file>`. (A full `generate --overwrite` rewrites everything, so save it for first setup.)
+
+A few things worth knowing:
+
+- **No config needed** — by default each spec pairs with the code file of the same name beside it (`parser.md` ↔ `parser.py`).
+  Prefer one spec per folder? `--layout per-dir` ([FAQ](FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
+- **Any language, three providers** — swap the model with `--model openai:…`, `google:…`, or `claude-code`
+  (Anthropic is the default); code is read as plain text. Want another provider? [Open an issue](#feedback).
+- **Your other docs are safe** — by default only a `.md` next to code with the same name counts as a spec, so your
+  READMEs are left alone. (Folder specs and overview indexes only appear if you opt in with `--layout` / `--overview`.)
+- **What's shared** — to grade a spec, its code and text go to your AI provider. That's it. (`coverage` sends nothing — it runs fully on your machine.)
+
+**Keep it honest on every commit** — add one line to a git hook (a script git runs before each commit) so new
+code always needs a spec; copy-paste setup in [the tutorial](GETTING-STARTED.md#make-it-routine):
+```bash
+spec-eval coverage . --min 90    # fails the commit if coverage drops below 90%  (free — no AI)
+```
+
+## How the scores are made
+
+Three scores, three simple ideas:
+
+- **coverage** = spec files you *have* ÷ spec files you *need* → a percent. Just counting, **no AI**.
+  *(9 of 10 code files have a spec → 90%.)*
+- **drift** = the number of spots where the spec and the code flat-out disagree. **`0` = they match.**
+- **sufficiency** = how much of what the code does is actually written in the spec, from **`0` to `1`**.
+  *(`1.0` = it's all there; `0.1` = almost none of it is.)*
+
+Those three are **per spec**. The repo's one **sufficiency** score is just their **average** — each spec's
+`0`–`1`, added up and divided by how many specs were scored. *(coverage is already one repo-wide `%`; drift is
+the total count across specs.)*
+
+`coverage` is pure counting. The other two need an **AI reader**: it reads each spec next to its code and grades
+the pair against a fixed rubric — one model call per pair. Two things follow from that:
+
+> [!NOTE]
+> AI scores **wobble**: the same spec might get 0.78 one run and 0.72 the next, like two teachers grading the same
+> essay. Watch the **trend**, not the decimal — a jump like 0.72 → 0.85 is a real improvement.
+
+> [!IMPORTANT]
+> AI calls **spend tokens** — through your API key, your Claude subscription with `--model claude-code`, or
+> your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.
+
+**Want the exact rules behind each score?** spec-eval specs its own scoring — read
+[`spec_eval/coverage.md`](spec_eval/coverage.md) (how coverage counts),
+[`spec_eval/rubric.md`](spec_eval/rubric.md) (the drift rules), and
+[`spec_eval/sufficiency.md`](spec_eval/sufficiency.md) (how the 0–1 score is decided).
+
+## Reading the scores
+
+Each run writes a short report to `spec-reports/` (a `.md` you read, a `.json` for tools):
+
+- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](spec-reports/coverage.md))*
+- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](spec-reports/report.md))*
+- **sufficiency** — a `0`–`1` score per spec (worst first), listing what's missing with a searchable code pointer
+  (`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](spec-reports/sufficiency.md))*
+
+The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](SPEC-HEALTH.md).
+
+Every command also appends a line to `runs.jsonl` — timestamp, git commit, scores — so you can track change over
+time.
+
+## Higher stakes? Get a second opinion
+
+`drift` and `sufficiency` are **AI judgments** — one model can miss something or be too generous. For a spec that needs extra attention — a critical path, complex design, or compliance code — run the
+**same check with two different AI vendors** and compare. Give each run its own
+`--out` folder (otherwise the second run overwrites the first report):
+
+```bash
+spec-eval audit ./your-project --model anthropic:claude-opus-4-8 --out spec-reports/claude
+spec-eval audit ./your-project --model google:gemini-3.5-flash   --out spec-reports/gemini
+
+diff spec-reports/claude/report.md spec-reports/gemini/report.md   # or read them side by side
+```
+
+How to read two reports:
+
+- **Both flag it** → almost certainly real. Fix these first.
+- **Only one flags it** → a judgment call. Read the quoted evidence and decide.
+- **Sufficiency scores differ a little** → normal wobble. Compare the **gaps lists**, not the decimals — the same
+  missing behavior named by both graders is the signal.
+
+Each vendor reads its own key from the environment — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY` —
+so put both keys in one `.env` and each run picks up the one it needs. (Want one fewer key? Swap either side for
+`--model claude-code`, which uses your Claude Code login instead.)
+
+## Proof it works
+
+- **spec-eval specs itself** — every module in [`spec_eval/`](spec_eval/) has its own spec:
+  **coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](SPEC-HEALTH.md) — the exact, dated score).
+
+## More
+
+- **Works with [spec-driven development](https://en.wikipedia.org/wiki/Spec-driven_development).** SDD's goal is a spec you can *trust*, and **spec-eval** is that trust layer.
+  Use it three ways: **grade the specs you already have**, **generate specs** for code that has none, or **check as
+  you write** (with the skills). Spec-first, code-first, or in between — it meets you where you are.
+- **Grow the spec and the code together.** The best version of "in between": draft a little spec, write a little
+  code, check they still agree — repeat. The two can even move in parallel (you shape the spec while your coding
+  agent writes the code; the check is where they meet), so the spec is *born accurate* instead of written up
+  afterward.
+- **Just want to vibe code?** Fine — build first, check later, automatically: a pre-commit gate and a GitHub
+  Action can run the checks for you — see [Make it routine](GETTING-STARTED.md#make-it-routine).
+- **Higher-stakes spec?** Run it past two vendors and compare — see [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion) above.
+
+## FAQ
+
+Common questions — from *"which command do I run first?"* through costs, layouts, and score accuracy — live in
+**[FAQ.md](FAQ.md)**.
+
+## Feedback
+
+Found a bug, a bad score, or a spec that misses the point? [Open an issue](https://github.com/benjaminjones/spec-eval/issues) — a report snippet and the command you ran is plenty.
+
+## Acknowledgements
+
+Built on the Anthropic, OpenAI, and Google SDKs (MIT / Apache-2.0) + PyYAML.
+
+---
+*Terms you may be searching for: software specification · documentation drift · living documentation · docs as code · backfill
+documentation · spec-driven development (SDD) · agentic development · AI-assisted coding · vibe coding · vibe engineering ·
+AI code documentation · LLM-as-judge / trust eval. MIT licensed.*

spec_eval-0.2.0/README.md

@@ -0,0 +1,264 @@
+# spec-eval — Build fast with AI. Validate with specs.
+
+**Your code changes fast. Your docs fall behind. spec-eval keeps them honest.**
+
+*The trust layer for [spec-driven development](https://en.wikipedia.org/wiki/Spec-driven_development) — iterate fast, keep the spec and code in agreement.*
+
+*A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.*
+
+[Get started](#try-it) · [Tutorial](GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](FAQ.md)
+
+A **spec** is short for **specification** — a plain-English description of what your code should do. **spec-eval**
+keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth
+trusting. How far you lean on it is up to you. **No specs yet? spec-eval writes them for you.**
+
+## Table of contents
+
+- [The problem](#the-problem)
+- [What it checks](#what-it-checks)
+- [What it creates](#what-it-creates)
+- [Try it](#try-it)
+  - [Run via prompt chat (no setup)](#run-via-prompt-chat-no-setup)
+  - [Run in the terminal, no key (Claude subscription)](#run-in-the-terminal-no-key-claude-subscription)
+  - [Run in the terminal with an API key (for CI)](#run-in-the-terminal-with-an-api-key-for-ci)
+- [How the scores are made](#how-the-scores-are-made)
+- [Reading the scores](#reading-the-scores)
+- [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion)
+- [Proof it works](#proof-it-works)
+- [More](#more)
+- [FAQ](#faq)
+- [Feedback](#feedback)
+- [Acknowledgements](#acknowledgements)
+
+## The problem
+
+Specs go stale the moment code moves on — and one stale spec is enough to stop a reader trusting all of them.
+AI coding agents make code move faster than any hand-maintained doc can follow.
+
+**spec-eval** treats specs like code: **run a check — via a chat prompt, an ordinary command, or a CI step — and
+the spec is measured against the source.** A spec you can check is a spec you can trust.
+
+## What it checks
+
+Three simple questions:
+
+| Check | The question it answers | How |
+|---|---|---|
+| **coverage** | Does this code have a spec at all? | Just counts files. Free — no AI, no key. |
+| **drift** | Does the spec say something the code doesn't do? | An AI reads each spec and its code and lists the clashes. |
+| **sufficiency** | How completely does the spec capture the code's behavior? | An AI lists what the spec misses and scores it 0–1 — an indicator, not a guarantee. |
+
+Your **code is the source of truth** — the spec is what gets graded against it.
+
+## What it creates
+
+Everything it writes is plain markdown, right next to your code:
+
+```text
+ your code                 specs beside it              spec-reports/
+┌────────────┐             ┌────────────┐             ┌────────────────┐
+│ parser.py  │  generate   │ parser.md  │  coverage   │ coverage.md    │
+│ api.py     │ ──────────► │ api.md     │  audit      │ report.md      │
+│ …          │             │ …          │ ──────────► │ sufficiency.md │
+└────────────┘             └────────────┘ sufficiency └───────┬────────┘
+                                                              │
+                                                              ▼
+                                                       SPEC-HEALTH.md
+                                                    (one-page scorecard)
+```
+
+## Try it
+
+Prefer a guided walkthrough? **[Getting-started tutorial →](GETTING-STARTED.md)**
+
+**The commands** — in the order you'll typically use them:
+
+- **coverage** — which files have no spec yet? *(free — no AI)*
+- **generate** — writes a plain-English spec beside each file that has none *(uses AI)*
+- **sufficiency** — how completely does each spec capture the code? Scored 0–1 *(uses AI)*
+- **audit** — the drift check: does any spec clash with the code? Run it at every milestone *(uses AI)*
+
+None of them change your files — reports go to `spec-reports/`. The exception is `generate`, which writes new
+specs beside your code: review them and commit the keepers like code.
+
+The `(uses AI)` commands (`generate`, `sufficiency`, `audit`) need an AI behind them — `coverage` never does.
+Three ways to run them, least setup first:
+
+### Run via prompt chat (no setup)
+
+Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
+land in the chat:
+```text
+Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
+first check coverage, then check my specs against my code.
+```
+The path is wherever you cloned this repo — no clone? Give your agent [the file's link](skills/spec-check/SKILL.md).
+
+Optionally save reports — add:
+```text
+and save the results to spec-reports/
+```
+The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).
+
+**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](skills/) into your agent's skills
+folder (Claude Code: `.claude/skills/`). Then the prompt is the command:
+```bash
+mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/
+```
+```text
+Which files have no spec yet?        # coverage
+Write a spec for src/parser.py       # generate
+Check my specs against my code       # sufficiency + audit
+```
+
+### Run in the terminal, no key (Claude subscription)
+
+Install once, then add `--model claude-code` to any `(uses AI)` command — **spec-eval** routes through the
+`claude` CLI you're already logged into (no key; your `ANTHROPIC_API_KEY` is hidden from the call, so it can't
+accidentally bill the paid API). The same four commands, in the same order as the list above:
+```bash
+pip install spec-eval                                     # or, from a clone:  pip install -e .
+spec-eval coverage    ./your-project                      # free — no AI, no key
+spec-eval generate    ./your-project --model claude-code
+spec-eval sufficiency ./your-project --model claude-code
+spec-eval audit       ./your-project --model claude-code
+```
+Every terminal run writes its reports to `spec-reports/` (plus a `runs.jsonl` history line).
+
+### Run in the terminal with an API key (for CI)
+
+Export a key and run any command above **without** `--model claude-code` — an API key is the default. This is
+the option for CI, the checks that run automatically on every push (store the key as a repo secret):
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...     # or OPENAI_API_KEY / GOOGLE_API_KEY
+```
+
+> [!NOTE]
+> A typical first run is `coverage → generate → sufficiency`, then at every milestone after: `audit` — plus
+> `sufficiency` when you've added behavior. The [tutorial](GETTING-STARTED.md) has step-by-step guides for
+> starting with no specs and for existing docs.
+
+> [!TIP]
+> **Version control is also your preview.** The three checks never modify your project; `generate` adds
+> ordinary new files — review them with `git diff`, edit what's off, and drop any you don't want with
+> `git checkout -- <file>`. (A full `generate --overwrite` rewrites everything, so save it for first setup.)
+
+A few things worth knowing:
+
+- **No config needed** — by default each spec pairs with the code file of the same name beside it (`parser.md` ↔ `parser.py`).
+  Prefer one spec per folder? `--layout per-dir` ([FAQ](FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
+- **Any language, three providers** — swap the model with `--model openai:…`, `google:…`, or `claude-code`
+  (Anthropic is the default); code is read as plain text. Want another provider? [Open an issue](#feedback).
+- **Your other docs are safe** — by default only a `.md` next to code with the same name counts as a spec, so your
+  READMEs are left alone. (Folder specs and overview indexes only appear if you opt in with `--layout` / `--overview`.)
+- **What's shared** — to grade a spec, its code and text go to your AI provider. That's it. (`coverage` sends nothing — it runs fully on your machine.)
+
+**Keep it honest on every commit** — add one line to a git hook (a script git runs before each commit) so new
+code always needs a spec; copy-paste setup in [the tutorial](GETTING-STARTED.md#make-it-routine):
+```bash
+spec-eval coverage . --min 90    # fails the commit if coverage drops below 90%  (free — no AI)
+```
+
+## How the scores are made
+
+Three scores, three simple ideas:
+
+- **coverage** = spec files you *have* ÷ spec files you *need* → a percent. Just counting, **no AI**.
+  *(9 of 10 code files have a spec → 90%.)*
+- **drift** = the number of spots where the spec and the code flat-out disagree. **`0` = they match.**
+- **sufficiency** = how much of what the code does is actually written in the spec, from **`0` to `1`**.
+  *(`1.0` = it's all there; `0.1` = almost none of it is.)*
+
+Those three are **per spec**. The repo's one **sufficiency** score is just their **average** — each spec's
+`0`–`1`, added up and divided by how many specs were scored. *(coverage is already one repo-wide `%`; drift is
+the total count across specs.)*
+
+`coverage` is pure counting. The other two need an **AI reader**: it reads each spec next to its code and grades
+the pair against a fixed rubric — one model call per pair. Two things follow from that:
+
+> [!NOTE]
+> AI scores **wobble**: the same spec might get 0.78 one run and 0.72 the next, like two teachers grading the same
+> essay. Watch the **trend**, not the decimal — a jump like 0.72 → 0.85 is a real improvement.
+
+> [!IMPORTANT]
+> AI calls **spend tokens** — through your API key, your Claude subscription with `--model claude-code`, or
+> your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.
+
+**Want the exact rules behind each score?** spec-eval specs its own scoring — read
+[`spec_eval/coverage.md`](spec_eval/coverage.md) (how coverage counts),
+[`spec_eval/rubric.md`](spec_eval/rubric.md) (the drift rules), and
+[`spec_eval/sufficiency.md`](spec_eval/sufficiency.md) (how the 0–1 score is decided).
+
+## Reading the scores
+
+Each run writes a short report to `spec-reports/` (a `.md` you read, a `.json` for tools):
+
+- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](spec-reports/coverage.md))*
+- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](spec-reports/report.md))*
+- **sufficiency** — a `0`–`1` score per spec (worst first), listing what's missing with a searchable code pointer
+  (`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](spec-reports/sufficiency.md))*
+
+The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](SPEC-HEALTH.md).
+
+Every command also appends a line to `runs.jsonl` — timestamp, git commit, scores — so you can track change over
+time.
+
+## Higher stakes? Get a second opinion
+
+`drift` and `sufficiency` are **AI judgments** — one model can miss something or be too generous. For a spec that needs extra attention — a critical path, complex design, or compliance code — run the
+**same check with two different AI vendors** and compare. Give each run its own
+`--out` folder (otherwise the second run overwrites the first report):
+
+```bash
+spec-eval audit ./your-project --model anthropic:claude-opus-4-8 --out spec-reports/claude
+spec-eval audit ./your-project --model google:gemini-3.5-flash   --out spec-reports/gemini
+
+diff spec-reports/claude/report.md spec-reports/gemini/report.md   # or read them side by side
+```
+
+How to read two reports:
+
+- **Both flag it** → almost certainly real. Fix these first.
+- **Only one flags it** → a judgment call. Read the quoted evidence and decide.
+- **Sufficiency scores differ a little** → normal wobble. Compare the **gaps lists**, not the decimals — the same
+  missing behavior named by both graders is the signal.
+
+Each vendor reads its own key from the environment — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_API_KEY` —
+so put both keys in one `.env` and each run picks up the one it needs. (Want one fewer key? Swap either side for
+`--model claude-code`, which uses your Claude Code login instead.)
+
+## Proof it works
+
+- **spec-eval specs itself** — every module in [`spec_eval/`](spec_eval/) has its own spec:
+  **coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](SPEC-HEALTH.md) — the exact, dated score).
+
+## More
+
+- **Works with [spec-driven development](https://en.wikipedia.org/wiki/Spec-driven_development).** SDD's goal is a spec you can *trust*, and **spec-eval** is that trust layer.
+  Use it three ways: **grade the specs you already have**, **generate specs** for code that has none, or **check as
+  you write** (with the skills). Spec-first, code-first, or in between — it meets you where you are.
+- **Grow the spec and the code together.** The best version of "in between": draft a little spec, write a little
+  code, check they still agree — repeat. The two can even move in parallel (you shape the spec while your coding
+  agent writes the code; the check is where they meet), so the spec is *born accurate* instead of written up
+  afterward.
+- **Just want to vibe code?** Fine — build first, check later, automatically: a pre-commit gate and a GitHub
+  Action can run the checks for you — see [Make it routine](GETTING-STARTED.md#make-it-routine).
+- **Higher-stakes spec?** Run it past two vendors and compare — see [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion) above.
+
+## FAQ
+
+Common questions — from *"which command do I run first?"* through costs, layouts, and score accuracy — live in
+**[FAQ.md](FAQ.md)**.
+
+## Feedback
+
+Found a bug, a bad score, or a spec that misses the point? [Open an issue](https://github.com/benjaminjones/spec-eval/issues) — a report snippet and the command you ran is plenty.
+
+## Acknowledgements
+
+Built on the Anthropic, OpenAI, and Google SDKs (MIT / Apache-2.0) + PyYAML.
+
+---
+*Terms you may be searching for: software specification · documentation drift · living documentation · docs as code · backfill
+documentation · spec-driven development (SDD) · agentic development · AI-assisted coding · vibe coding · vibe engineering ·
+AI code documentation · LLM-as-judge / trust eval. MIT licensed.*

spec_eval-0.2.0/pyproject.toml

@@ -0,0 +1,52 @@
+[project]
+name = "spec-eval"
+dynamic = ["version"]                     # single source of truth: spec_eval/__init__.py (__version__)
+description = "A trust-eval for your specs: coverage, drift, and sufficiency of markdown specs against code."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Ben Jones" }]
+keywords = ["spec-driven-development", "documentation-drift", "docs-as-code", "living-documentation",
+            "llm-as-judge", "code-documentation", "backfill-documentation"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Documentation",
+    "Topic :: Software Development :: Quality Assurance",
+]
+dependencies = [
+    "anthropic>=0.40",
+    "openai>=1.50",
+    "google-genai>=1.0",
+    "PyYAML>=6.0",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7", "hypothesis>=6", "ruff>=0.4"]
+
+[project.scripts]
+spec-eval = "spec_eval.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/benjaminjones/spec-eval"
+Repository = "https://github.com/benjaminjones/spec-eval"
+Issues = "https://github.com/benjaminjones/spec-eval/issues"
+
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = ["spec_eval"]
+
+[tool.setuptools.dynamic]
+version = { attr = "spec_eval.__version__" }
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+
+[tool.ruff]                       # Layer 0 — static analysis
+line-length = 120

spec_eval-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

spec_eval-0.2.0/spec_eval/__init__.py

@@ -0,0 +1,2 @@
+"""spec-eval — portable code↔doc drift audit + fingerprint."""
+__version__ = "0.2.0"   # THE single source of version truth; pyproject.toml reads this via dynamic version

spec_eval-0.2.0/spec_eval/__main__.py

@@ -0,0 +1,4 @@
+from .cli import main
+
+if __name__ == "__main__":
+    main()