specproof 0.2.0 → 0.3.0

package/CLAUDE.md

@@ -1,82 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## What this is
-
-SpecProof: a standalone Next.js app that renders an audit view of any repo's API test coverage: every OpenAPI operation and response status, cross-examined against the target repo's test assertions. Clicking a stamped verdict shows the actual test snippet that proves (or fails to prove) coverage.
-
-This repo does not contain a real API or its tests — it audits a target repo from the outside, the way OpenAPI tooling works when installed into a repo. Nothing repo-specific is baked in. It does carry a hand-written demo target in `example/` (see "Example fixture" below), which is what gets audited when no `SPECPROOF_REPO` is set.
-
-## Architecture: generate → checked-in artifact → contract-test
-
-1. **Sources of truth**, read from the target repo (`SPECPROOF_REPO` env var; defaults to the current working directory):
-   - The OpenAPI spec — auto-discovered as the shallowest `openapi*.json` / `swagger*.json` in the tree, or set explicitly with `SPECPROOF_SPEC` (relative to the repo root).
-   - The repo's `*.test.ts` / `*.test.tsx` / `*.test.js` files — parsed via regex for `describe("METHOD /path")` blocks and `.status).toBe(NNN)` / `.status).toEqual(NNN)` assertions.
-2. **Analyzer** (`lib/api-test-coverage.ts`): joins the two sources into a `CoverageReport` (tags → operations → statuses, each status with assertion counts and extracted `it()` snippets). Operations are joined on the describe title's method + path; `{param}`, `[param]`, and `:param` segments are treated as equivalent. When several test files describe the same operation, the one with the most `it()` blocks wins.
-3. **Generator** (`scripts/generate-proof.ts`): calls the analyzer, writes the result to `app/proof.generated.json`. This is checked in, deterministic, and regenerated automatically before `dev`/`build`. When no target spec is found it leaves the existing artifact untouched (exit 0), so the app always builds.
-4. **Consumer** (`app/page.tsx` → `components/CoverageProof.tsx`): renders the checked-in artifact only. No target checkout is needed to build or deploy the app itself. An empty proof renders an empty-state hint.
-5. **Drift guard** (`app/proof-contract.test.ts`): fails when the artifact is stale relative to the spec/tests, when the proof's audited operations don't exactly match the spec's operations, or when a quoted snippet no longer points at a real `it()`/`test()` line. This suite self-skips (via `describe.skipIf`) when no target spec is resolvable — in this repo one always is (the example fixture), so `bun run test:unit` always exercises the full analyzer here.
-
-## Example fixture (`example/`)
-
-`example/` is a hand-written demo target: `example/api/openapi.json` (the fictional "TaskFlow" API, 7 operations) plus `example/tests/*.test.ts` written in exactly the `describe("METHOD /path")` / `.status).toBe(NNN)` shape the parser expects. Because spec auto-discovery finds the shallowest `openapi*.json` under the target root (cwd by default), running `dev` / `generate:proof` / `test:unit` from this repo audits the example automatically — no `SPECPROOF_REPO` needed. When the package is installed into another repo, the example sits under `node_modules/`, which the scanner excludes, so it never pollutes a real audit.
-
-- The example files are **fixture data parsed as text**, never executed or compiled: they are excluded from vitest (`vitest.config.ts`), tsc (`tsconfig.json`), and eslint (`eslint.config.mjs`). They only need to look like a prettier-formatted test suite.
-- The spec and tests are **deliberately out of step** so the demo shows every verdict state: fully proven operations (`POST /auth/login`, `GET /tasks`, `GET /tasks/{taskId}`), partial coverage (`POST /tasks`, `PATCH /tasks/{taskId}`), untested operations (`DELETE /tasks/{taskId}`, `GET /projects`), and one status the tests assert but the spec omits (422 on `POST /tasks`). Preserve that mix when editing.
-- Editing anything in `example/` makes the checked-in proof stale and fails the contract tests: run `bun run generate:proof` and commit the regenerated `app/proof.generated.json`.
-
-Because of this split, most day-to-day work happens in exactly one of two places:
-- Changing what's audited/how coverage is computed → `lib/api-test-coverage.ts` (+ regenerate the proof).
-- Changing how the proof is displayed → `components/CoverageProof.tsx` / `app/proof.css`, reading the existing `app/proof.generated.json` as fixture data — no target checkout needed.
-
-## Commands
-
-Requires [Bun](https://bun.sh).
-
-```bash
-bun install
-bun run dev              # generate:proof + next dev (port 3001)
-bun run build            # generate:proof + next build
-bun run start            # serve the production build (port 3001)
-bun run generate:proof   # regenerate app/proof.generated.json only
-bun run test:unit        # contract tests (vitest run; audits example/ unless SPECPROOF_REPO is set)
-bun run test:unit:watch  # vitest watch mode
-bun run lint             # eslint + tsc --noEmit
-bun run lint:es          # eslint only
-bun run lint:ts          # tsc --noEmit only
-```
-
-To run a single test file: `bunx vitest run app/proof-contract.test.ts` (contract tests) or `bunx vitest run lib/api-test-coverage.test.ts` (analyzer unit tests). The unit tests' inline fixtures deliberately use operations that don't exist in the example spec (`/widgets`, `/gadgets`) — the analyzer parses this repo's own `*.test.ts` files as raw text when auditing `example/`, and colliding paths would pollute the generated proof.
-
-To audit a real repo instead of the bundled example:
-
-```bash
-SPECPROOF_REPO=/path/to/target-repo bun run generate:proof
-# optionally, if the spec has a nonstandard name/location:
-SPECPROOF_REPO=/path/to/target-repo SPECPROOF_SPEC=docs/api-spec.json bun run generate:proof
-```
-
-## Updating the proof
-
-When a target repo's test or OpenAPI spec changes, the checked-in `app/proof.generated.json` goes stale and the contract tests fail. Fix:
-
-```bash
-SPECPROOF_REPO=/path/to/target-repo bun run generate:proof
-git add app/proof.generated.json && git commit
-```
-
-## CI & Releasing
-
-`main` is protected by a repo ruleset (PR required, merge queue, no direct pushes). The flow: open a PR → the CI workflow (`.github/workflows/test-unit.yml`, jobs `Build` / `Lint` / `Test`) runs on the PR and again in the merge queue → all three checks must pass → clicking merge queues the PR and GitHub merges it (squash).
-
-Every merge to `main` then triggers `.github/workflows/release.yml`, which publishes to npm when `package.json`'s version isn't already on the registry (needs the `NPM_TOKEN` repo secret). To release: bump `version` in `package.json`, open a PR, and merge. There is no manual release trigger.
-
-## Notes on the parsing approach
-
-`lib/api-test-coverage.ts` parses test files with regexes rather than a real TS/AST parser, relying on prettier-consistent formatting conventions in the audited repo:
-- `describe("METHOD /path")` titles are matched for the HTTP method prefix (`GET|POST|PUT|DELETE|PATCH`) followed by the operation path.
-- `it()`/`test()` blocks are extracted by matching the opening call, then scanning for the next `});` at the same indentation level — not a brace-matching parser, so it depends on consistent formatting in the audited repo.
-- Snippet source is dedented before being stored/rendered.
-
-If this parsing logic needs updating, check it against real test files in a target repo — the contract test's "every quoted snippet points at a real it()/test() block" check is what catches regressions here.

package/app/proof-contract.test.ts

@@ -1,95 +0,0 @@
-import fs from "fs";
-import path from "path";
-import { describe, expect, it } from "vitest";
-
-import {
-  resolveSpecPath,
-  TARGET_REPO_ROOT,
-  type CoverageReport,
-} from "@/lib/api-test-coverage";
-import { buildProof, GENERATED_PROOF_PATH } from "@/scripts/generate-proof";
-
-/**
- * Contract tests that keep the rendered proof in lockstep with the target
- * repo it audits. Three drift modes are covered:
- *
- * 1. The OpenAPI spec gains/loses an operation without the proof following
- *    (spec <-> proof, both directions).
- * 2. A quoted test snippet no longer points at a real it()/test() block in
- *    its test file — the evidence the UI shows must be re-readable from the
- *    source it cites.
- * 3. The spec or a test file changed without regenerating
- *    proof.generated.json.
- */
-
-function loadCheckedInProof(): CoverageReport {
-  return JSON.parse(fs.readFileSync(GENERATED_PROOF_PATH, "utf8"));
-}
-
-// The drift guards compare the artifact against the audited target repo
-// (SPECPROOF_REPO, or the current directory). Without an OpenAPI spec to audit
-// there is nothing to compare against, so the suite self-skips.
-const specPath = resolveSpecPath();
-if (!specPath) {
-  console.warn(
-    `proof-contract: skipping — no OpenAPI spec found under ${TARGET_REPO_ROOT} (set SPECPROOF_REPO to the repo to audit, or SPECPROOF_SPEC to the spec file)`,
-  );
-}
-
-describe.skipIf(!specPath)("SpecProof contract", () => {
-  it("audits every operation in the OpenAPI spec, and nothing else", () => {
-    const spec = JSON.parse(fs.readFileSync(specPath!, "utf8")) as {
-      paths: Record<string, Record<string, unknown>>;
-    };
-
-    const documented = Object.entries(spec.paths)
-      .flatMap(([opPath, methods]) =>
-        Object.keys(methods).map((method) => `${method} ${opPath}`),
-      )
-      .sort();
-
-    const audited = loadCheckedInProof()
-      .tags.flatMap((tag) => tag.operations)
-      .map((op) => `${op.method} ${op.specPath}`)
-      .sort();
-
-    expect(documented.length).toBeGreaterThan(0);
-    expect(audited).toEqual(documented);
-  });
-
-  it("every quoted snippet points at a real it()/test() block in its test file", () => {
-    const sources = new Map<string, string[]>();
-    const readLines = (testFile: string) => {
-      const cached = sources.get(testFile);
-      if (cached) return cached;
-      const lines = fs
-        .readFileSync(path.join(TARGET_REPO_ROOT, testFile), "utf8")
-        .split("\n");
-      sources.set(testFile, lines);
-      return lines;
-    };
-
-    for (const tag of loadCheckedInProof().tags) {
-      for (const op of tag.operations) {
-        if (!op.testFile) continue;
-        for (const status of op.statuses) {
-          for (const snippet of status.snippets) {
-            const line = readLines(op.testFile)[snippet.startLine - 1] ?? "";
-            expect(
-              /(?:it|test)\(/.test(line),
-              `${op.method} ${op.specPath} ${status.code}: snippet "${snippet.title}" cites ${op.testFile}:${snippet.startLine}, but that line is not an it()/test() block`,
-            ).toBe(true);
-          }
-        }
-      }
-    }
-  });
-
-  it("proof.generated.json is up to date with the spec and tests", () => {
-    const checkedIn = loadCheckedInProof();
-    // If this fails, the OpenAPI spec or a test file changed without
-    // regenerating the proof: run `bun run generate:proof` and commit the
-    // result.
-    expect(checkedIn).toEqual(buildProof());
-  });
-});

package/app/proof.generated.json

@@ -1,288 +0,0 @@
-{
-  "tags": [
-    {
-      "tag": "Auth",
-      "description": "Session and token management",
-      "operations": [
-        {
-          "method": "post",
-          "specPath": "/auth/login",
-          "summary": "Exchange credentials for a session token",
-          "testFile": "example/tests/auth.test.ts",
-          "testCount": 2,
-          "statuses": [
-            {
-              "code": "200",
-              "description": "Session token issued",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "returns a session token for valid credentials",
-                  "source": "it(\"returns a session token for valid credentials\", async () => {\n  const res = await api.post(\"/auth/login\", {\n    email: \"ada@example.com\",\n    password: \"correct horse battery staple\",\n  });\n\n  expect(res.status).toBe(200);\n  expect(res.body.token).toMatch(/^tk_/);\n});",
-                  "startLine": 6
-                }
-              ]
-            },
-            {
-              "code": "401",
-              "description": "Invalid credentials",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "rejects invalid credentials",
-                  "source": "it(\"rejects invalid credentials\", async () => {\n  const res = await api.post(\"/auth/login\", {\n    email: \"ada@example.com\",\n    password: \"not-the-password\",\n  });\n\n  expect(res.status).toBe(401);\n  expect(res.body.token).toBeUndefined();\n});",
-                  "startLine": 16
-                }
-              ]
-            }
-          ],
-          "coveredCount": 2,
-          "gapCount": 0
-        }
-      ],
-      "coveredCount": 2,
-      "totalCount": 2
-    },
-    {
-      "tag": "Tasks",
-      "description": "Create and manage tasks",
-      "operations": [
-        {
-          "method": "get",
-          "specPath": "/tasks",
-          "summary": "List tasks for the authenticated user",
-          "testFile": "example/tests/tasks.test.ts",
-          "testCount": 2,
-          "statuses": [
-            {
-              "code": "200",
-              "description": "Task list",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "lists tasks for the authenticated user",
-                  "source": "it(\"lists tasks for the authenticated user\", async () => {\n  const res = await api.get(\"/tasks\");\n\n  expect(res.status).toBe(200);\n  expect(Array.isArray(res.body.tasks)).toBe(true);\n});",
-                  "startLine": 14
-                }
-              ]
-            },
-            {
-              "code": "401",
-              "description": "Missing or invalid bearer token",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "rejects requests without a bearer token",
-                  "source": "it(\"rejects requests without a bearer token\", async () => {\n  const res = await api.get(\"/tasks\", { auth: false });\n\n  expect(res.status).toBe(401);\n});",
-                  "startLine": 21
-                }
-              ]
-            }
-          ],
-          "coveredCount": 2,
-          "gapCount": 0
-        },
-        {
-          "method": "post",
-          "specPath": "/tasks",
-          "summary": "Create a task",
-          "testFile": "example/tests/tasks.test.ts",
-          "testCount": 3,
-          "statuses": [
-            {
-              "code": "201",
-              "description": "Task created",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "creates a task and returns it",
-                  "source": "it(\"creates a task and returns it\", async () => {\n  const res = await api.post(\"/tasks\", {\n    title: \"Ship the Q3 roadmap\",\n    projectId: \"proj_1\",\n  });\n\n  expect(res.status).toBe(201);\n  expect(res.body.id).toBeDefined();\n  expect(res.body.title).toBe(\"Ship the Q3 roadmap\");\n});",
-                  "startLine": 29
-                }
-              ]
-            },
-            {
-              "code": "400",
-              "description": "Malformed request body",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "rejects a task without a title",
-                  "source": "it(\"rejects a task without a title\", async () => {\n  const res = await api.post(\"/tasks\", { projectId: \"proj_1\" });\n\n  expect(res.status).toBe(400);\n  expect(res.body.error).toBe(\"title is required\");\n});",
-                  "startLine": 40
-                }
-              ]
-            },
-            {
-              "code": "401",
-              "description": "Missing or invalid bearer token",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            },
-            {
-              "code": "422",
-              "description": "",
-              "documented": false,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "rejects a due date in the past",
-                  "source": "it(\"rejects a due date in the past\", async () => {\n  const res = await api.post(\"/tasks\", {\n    title: \"Time travel\",\n    dueDate: \"1999-12-31\",\n  });\n\n  expect(res.status).toBe(422);\n});",
-                  "startLine": 47
-                }
-              ]
-            }
-          ],
-          "coveredCount": 2,
-          "gapCount": 1
-        },
-        {
-          "method": "get",
-          "specPath": "/tasks/{taskId}",
-          "summary": "Fetch a single task",
-          "testFile": "example/tests/tasks.test.ts",
-          "testCount": 2,
-          "statuses": [
-            {
-              "code": "200",
-              "description": "The task",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "returns a task by id",
-                  "source": "it(\"returns a task by id\", async () => {\n  const created = await api.post(\"/tasks\", { title: \"Write release notes\" });\n  const res = await api.get(`/tasks/${created.body.id}`);\n\n  expect(res.status).toBe(200);\n  expect(res.body.id).toBe(created.body.id);\n});",
-                  "startLine": 58
-                }
-              ]
-            },
-            {
-              "code": "404",
-              "description": "No task with this id",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "returns 404 for an unknown task id",
-                  "source": "it(\"returns 404 for an unknown task id\", async () => {\n  const res = await api.get(\"/tasks/task_does_not_exist\");\n\n  expect(res.status).toBe(404);\n});",
-                  "startLine": 66
-                }
-              ]
-            }
-          ],
-          "coveredCount": 2,
-          "gapCount": 0
-        },
-        {
-          "method": "patch",
-          "specPath": "/tasks/{taskId}",
-          "summary": "Update a task's fields",
-          "testFile": "example/tests/tasks.test.ts",
-          "testCount": 1,
-          "statuses": [
-            {
-              "code": "200",
-              "description": "Updated task",
-              "documented": true,
-              "assertions": 1,
-              "snippets": [
-                {
-                  "title": "updates the fields of a task",
-                  "source": "it(\"updates the fields of a task\", async () => {\n  const created = await api.post(\"/tasks\", { title: \"Refine the backlog\" });\n  const res = await api.patch(`/tasks/${created.body.id}`, {\n    title: \"Refine and prioritize the backlog\",\n  });\n\n  expect(res.status).toBe(200);\n  expect(res.body.title).toBe(\"Refine and prioritize the backlog\");\n});",
-                  "startLine": 74
-                }
-              ]
-            },
-            {
-              "code": "404",
-              "description": "No task with this id",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            },
-            {
-              "code": "409",
-              "description": "Task was modified concurrently",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            }
-          ],
-          "coveredCount": 1,
-          "gapCount": 2
-        },
-        {
-          "method": "delete",
-          "specPath": "/tasks/{taskId}",
-          "summary": "Delete a task",
-          "testFile": null,
-          "testCount": 0,
-          "statuses": [
-            {
-              "code": "204",
-              "description": "Task deleted",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            },
-            {
-              "code": "404",
-              "description": "No task with this id",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            }
-          ],
-          "coveredCount": 0,
-          "gapCount": 2
-        }
-      ],
-      "coveredCount": 7,
-      "totalCount": 12
-    },
-    {
-      "tag": "Projects",
-      "description": "Group tasks into projects",
-      "operations": [
-        {
-          "method": "get",
-          "specPath": "/projects",
-          "summary": "List projects",
-          "testFile": null,
-          "testCount": 0,
-          "statuses": [
-            {
-              "code": "200",
-              "description": "Project list",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            },
-            {
-              "code": "401",
-              "description": "Missing or invalid bearer token",
-              "documented": true,
-              "assertions": 0,
-              "snippets": []
-            }
-          ],
-          "coveredCount": 0,
-          "gapCount": 2
-        }
-      ],
-      "coveredCount": 0,
-      "totalCount": 2
-    }
-  ],
-  "operationCount": 7,
-  "coveredCount": 9,
-  "totalCount": 16,
-  "untestedOperations": 2
-}