@workerstack/testing 0.0.0-stage → 0.31.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 lovrozagar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,135 @@
-# Temporary Holding Version
+# @workerstack/testing
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![npm](https://img.shields.io/npm/v/@workerstack/testing.svg)](https://www.npmjs.com/package/@workerstack/testing)
+
+Testing for APIs, web apps and Workers. Today it ships `/api`, the API tester: live **matrix testing**
+of a running backend against its own OpenAPI document. The API tester reads the spec, talks to the API, and treats every way to see a record as a cell in a matrix: list,
+item, filter, sort, page, cursor, select, search, parent routes, another tenant. Then it checks that
+the cells agree. It never reads your source or assumes your framework.
+
+A schema fuzzer asks _did this answer match its schema?_ The API tester asks that of every request it sends,
+and then the questions only state can answer:
+
+- the row is on `GET /tables/{id}` and missing from `GET /tables`
+- `?filter=status.eq.nope` returns every row (the backend dropped the parameter)
+- `limit=2` walks to 9 rows and `limit=100` to 10 (the sort has no total order)
+- `PATCH { name }` also cleared `description`
+- `?filter=id.eq.<another tenant's id>` returns that tenant's row
+
+There is no ground-truth database. A filter and its negation must partition the set, a page walk
+must cover the collection, and a record read four ways must read the same. One root cause is one
+finding; the checks that depend on it are `BLOCKED`, not a page of copies.
+
+## Install
+
+```bash
+npm i -D @workerstack/testing
+```
+
+Requires **Node.js 22.18 or later**. The API tester reads OpenAPI 3.0–3.2 documents, in JSON or YAML, and
+runs as `testing api`. Renamed: [the migration guide](docs/api/migrating-from-oat.md) maps every name.
+
+## Quick start
+
+```bash
+testing api doctor --spec https://api.example.com/openapi.json   # offline: what the API tester can test, and what is missing
+testing api init --spec https://api.example.com/openapi.json     # writes testing.api.config.ts from the security schemes
+testing api run --config testing.api.config.ts                            # tests the running API
+```
+
+`doctor` names every coverage gap and the tag that would close it. To see the API tester find bugs first, run
+the bundled demo API with planted defects (`testing api serve --defects STALE_LIST,PATCH_REPLACES`) and
+point `labs/api/local.config.ts` at the URL it prints.
+
+## Authentication
+
+Each operation is sent the credentials its OpenAPI `security` requires, placed where its scheme
+says. The config only says how to get each scheme's value:
+
+```ts
+// testing.api.config.ts
+import { defineConfig } from "@workerstack/testing/api"
+
+export default defineConfig({
+	spec: "https://api.example.com/openapi.json",
+	baseUrl: "https://api.example.com",
+	security: { bearer: { static: "${API_TOKEN}" } },
+	principals: {
+		alpha: { tenant: "a" },
+		beta: { tenant: "b", security: { bearer: { static: "${API_TOKEN_B}" } } },
+	},
+})
+```
+
+| recipe   | the value                                                                         |
+| -------- | --------------------------------------------------------------------------------- |
+| `static` | a fixed token or key, or `{ username, password }` for basic auth                  |
+| `login`  | steps the API tester runs (a sign-in), then where the credential is in the answer |
+| `mint`   | an operation called with another scheme's credential (an API key)                 |
+| `oauth2` | a `clientCredentials` or `password` grant at the document's token URL             |
+| `hook`   | `hooks.resolveCredential` returns it                                              |
+
+A second principal in another `tenant` is what makes the isolation checks run.
+
+## Verdicts and exit codes
+
+| verdict        | meaning                                                           |
+| -------------- | ----------------------------------------------------------------- |
+| `BACKEND_BUG`  | the handler is wrong                                              |
+| `SPEC_BUG`     | the document is wrong, or disagrees with the handler              |
+| `SECURITY`     | an isolation or authorization failure across a declared boundary  |
+| `AMBIGUITY`    | the same evidence, where the boundary was only inferred           |
+| `COVERAGE_GAP` | a check could not run; the report names the tag that would let it |
+| `BLOCKED`      | a check this one depends on already failed                        |
+
+| exit | means                                                                            |
+| ---- | -------------------------------------------------------------------------------- |
+| `0`  | no defects                                                                       |
+| `1`  | a root-cause finding (`BACKEND_BUG`, `SPEC_BUG`, `SECURITY`, `AMBIGUITY`)        |
+| `2`  | usage: a missing or wrong flag, or a config the document contradicts             |
+| `3`  | The API tester could not do what it was asked (the network never came back, say) |
+| `4`  | the run may have left records behind; they are listed under Leftovers            |
+
+Any non-zero exit fails CI. Gaps and blocked checks never fail a full run on their own.
+
+## CI
+
+```yaml
+- run: npm i -D @workerstack/testing
+- run: npx testing api doctor --spec "$SPEC_URL"
+- run: npx testing api run --config testing.api.config.ts
+  env:
+    API_TOKEN: ${{ secrets.API_TOKEN }}
+    API_TOKEN_B: ${{ secrets.API_TOKEN_B }}
+```
+
+The report is written to `.testing/api/runs/latest/report.json`; read it to sort findings by verdict.
+
+## Limits
+
+- Bodies are generated from the document; the API tester checks HTTP statuses and JSON, never file contents.
+- A 429 is retried up to 5 times, honoring `Retry-After` up to `network.maxRetryAfterMs` (default 60s); a longer wait is not waited for, and the 429 stands.
+- Leftover rows on a shared database poison the numeric and filter checks: wipe them between runs.
+- Webhooks, callbacks and links are not followed, and URL `$ref`s are not fetched.
+
+The full list, with what you see and what to do instead, is in [docs/api/limits.md](docs/api/limits.md).
+
+## Docs
+
+- [Commands](docs/api/commands.md): `run`, `doctor`, `plan`, `init`, `serve`, `conformance`
+- [Configuration](docs/api/configuration/reference.md): every field; then [principals and security](docs/api/configuration/principals-and-security.md), [hooks](docs/api/configuration/hooks.md), [origins and uploads](docs/api/configuration/origins-and-uploads.md)
+- [Migrating to 0.25](docs/api/migrating-0.25.md): the removed config fields, and relative paths now read from the config's folder
+- [How the API tester reads a document](docs/api/model.md): how a run works, entities, seeding, query grammars
+- [OpenAPI tags](docs/api/tags.md): every `x-*` tag and what it unlocks
+- [Checks and verdicts](docs/api/checks.md): what each check needs and asserts
+- [Reports](docs/api/reports.md): `report.json`, `matrix.json`, repro scripts, progress logs
+- [Programmatic API](docs/api/api.md); [step builders](docs/api/steps.md): typed scenario steps from `@workerstack/testing/api/steps`
+- [Assumptions](docs/api/assumptions.md): what the API tester assumes, and how to declare an exception
+- [Migrating](docs/api/migrating-from-oat.md): every name the rename changed, old to new
+- [Limits](docs/api/limits.md): what the API tester cannot do, and what to do instead
+- [Reference defects](docs/api/reference-defects.md): the named lies `testing api serve --defects` can tell
+- [Releasing](docs/api/releasing.md): how a tag publishes to npm with provenance, and the steps to cut one
+
+## License
+
+MIT

package/bin/testing.js

@@ -0,0 +1,4 @@
+#!/usr/bin/env node
+import { start } from "../dist/cli.js"
+
+start()

package/dist/api/cli/commands/baseline.d.ts

@@ -0,0 +1,5 @@
+import { type Args } from "../flags.ts";
+import type { Writer } from "../writer.ts";
+/** `testing api baseline --from <run>`: the run's root-cause findings, as a baseline file. */
+export declare function commandBaseline(writer: Writer, flags: Args["flags"]): Promise<number>;
+//# sourceMappingURL=baseline.d.ts.map

package/dist/api/cli/commands/baseline.js

@@ -0,0 +1,46 @@
+import { existsSync } from "node:fs";
+import { readFile, realpath, writeFile } from "node:fs/promises";
+import { resolve } from "node:path";
+import { messageOf } from "../../errors/message.js";
+import { schemaRefusal } from "../../report/diff.js";
+import { baselineFrom } from "../../runtime/baseline.js";
+import { EXIT } from "../../runtime/exit.js";
+import { str } from "../flags.js";
+import { legacyRunHint } from "../legacy-run.js";
+/** `testing api baseline --from <run>`: the run's root-cause findings, as a baseline file. */
+export async function commandBaseline(writer, flags) {
+    const from = str(flags, "from");
+    if (from === undefined) {
+        writer.err("testing api: baseline needs --from <run folder | latest>\n");
+        return EXIT.usage;
+    }
+    const out = resolve(str(flags, "out") ?? "testing.api.baseline.json");
+    let saved;
+    try {
+        /* `latest` is a link: read where it points. */
+        const folder = await realpath(resolve(from));
+        saved = JSON.parse(await readFile(resolve(folder, "report.json"), "utf8"));
+    }
+    catch (error) {
+        writer.err(`testing api: ${legacyRunHint(resolve(from)) ?? `cannot read ${resolve(from, "report.json")}: ${messageOf(error)}`}\n`);
+        return EXIT.usage;
+    }
+    if (!Array.isArray(saved.findings)) {
+        writer.err(`testing api: ${resolve(from, "report.json")} holds no findings list\n`);
+        return EXIT.usage;
+    }
+    const refusal = schemaRefusal(saved, resolve(from, "report.json"));
+    if (refusal !== undefined) {
+        writer.err(`testing api: ${refusal}\n`);
+        return EXIT.usage;
+    }
+    if (flags.force !== true && existsSync(out)) {
+        writer.err(`testing api: ${out} exists; --force overwrites it\n`);
+        return EXIT.usage;
+    }
+    const baseline = baselineFrom(saved.findings);
+    await writeFile(out, `${JSON.stringify(baseline, null, "\t")}\n`);
+    writer.out(`  ${baseline.entries.length} known finding${baseline.entries.length === 1 ? "" : "s"} → ${out}\n`);
+    return 0;
+}
+//# sourceMappingURL=baseline.js.map

package/dist/api/cli/commands/conformance-matrix.d.ts

@@ -0,0 +1,40 @@
+import type { Writer } from "../writer.ts";
+import type { Toolkit } from "./conformance-toolkit.ts";
+export declare const BACKENDS: readonly ["memory", "sqlite", "postgres", "d1"];
+export type BackendName = (typeof BACKENDS)[number];
+/** The parts of the self-test, each runnable on its own: `--legs suites,passes,…`. */
+export declare const LEGS: readonly ["suites", "passes", "shapes", "recall", "scope", "fuzz"];
+export type Leg = (typeof LEGS)[number];
+/** One leg of the matrix: every defect against one storage engine in one API dialect. */
+export interface Pass {
+    backend: BackendName;
+    dialect: string;
+}
+/** What the matrix runs: its passes, its legs, and what the flags narrowed. */
+export interface Matrix {
+    passes: Pass[];
+    legs: ReadonlySet<Leg>;
+    /** Defect names `--only` kept; all when absent. */
+    only: string[] | undefined;
+    /** Shapes `--shapes` kept for the shapes and recall legs; all when absent. */
+    shapes: string[] | undefined;
+    jobs: number;
+}
+/**
+ * The legs to run. Backends vary the storage engine; dialects vary the API's conventions. The full
+ * cross product would be mostly redundant, so every backend runs the default dialect and the
+ * cheapest one runs the other shapes: `classic` renames everything, `linked` changes the pagination
+ * model, and `template` is a grammar the API tester knows only from its declaration.
+ */
+export declare function planPasses(backends: readonly BackendName[], requested: string | undefined, dialect: string | undefined): Pass[];
+/** The legs asked for, printed in order as each is answered. Returns what failed and what was proven. */
+export declare function runMatrix(writer: Writer, toolkit: Toolkit, matrix: Matrix): Promise<{
+    failures: number;
+    proven: Set<string>;
+}>;
+/** The checks clean recall cases prove: each recalled defect's primary check, the first it names. */
+export declare function recalledChecks(cases: ReadonlyArray<{
+    label: string;
+    problems: readonly string[];
+}>, expected: Readonly<Record<string, string | string[]>>): string[];
+//# sourceMappingURL=conformance-matrix.d.ts.map

package/dist/api/cli/commands/conformance-matrix.js

@@ -0,0 +1,193 @@
+export const BACKENDS = ["memory", "sqlite", "postgres", "d1"];
+/** The parts of the self-test, each runnable on its own: `--legs suites,passes,…`. */
+export const LEGS = ["suites", "passes", "shapes", "recall", "scope", "fuzz"];
+/**
+ * The legs to run. Backends vary the storage engine; dialects vary the API's conventions. The full
+ * cross product would be mostly redundant, so every backend runs the default dialect and the
+ * cheapest one runs the other shapes: `classic` renames everything, `linked` changes the pagination
+ * model, and `template` is a grammar the API tester knows only from its declaration.
+ */
+export function planPasses(backends, requested, dialect) {
+    if (dialect !== undefined || requested !== undefined) {
+        return backends.map((backend) => ({ backend, dialect: dialect ?? "postgrest" }));
+    }
+    return [
+        ...backends.map((backend) => ({ backend, dialect: "postgrest" })),
+        ...["classic", "linked", "jsonapi", "plain", "template"].map((extra) => ({
+            backend: "memory",
+            dialect: extra,
+        })),
+    ];
+}
+/** How many defects one task carries: a long leg spreads across the pool. */
+const CHUNK = 6;
+/** `items` in slices of `CHUNK`. */
+function chunked(items) {
+    return Array.from({ length: Math.ceil(items.length / CHUNK) }, (_, index) => items.slice(index * CHUNK, (index + 1) * CHUNK));
+}
+/** The requests a leg's task answers sent, summed source by source. */
+function requestsOf(toolkit, answers) {
+    return toolkit.suite.sumRequests(answers.map((answer) => answer.requests));
+}
+/** Seconds since `since`, to a tenth. */
+function seconds(since) {
+    return Math.round((performance.now() - since) / 100) / 10;
+}
+/** Every pass queued at once; its baselines and its defects in chunks, its results back in order. */
+function queuePasses(run) {
+    const { toolkit, pool, matrix } = run;
+    return matrix.passes.map(async (pass) => {
+        const defects = toolkit.suite.defectsFor(pass.backend, pass.dialect, matrix.only);
+        const answers = await Promise.all([
+            pool.run({ backend: pass.backend, dialect: pass.dialect, kind: "baselines" }),
+            ...chunked(defects).map((chunk) => pool.run({ backend: pass.backend, dialect: pass.dialect, defects: chunk, kind: "defects" })),
+        ]);
+        for (const answer of answers)
+            run.peakRss = Math.max(run.peakRss, answer.memory.rss);
+        const results = answers.flatMap((answer) => answer.results);
+        const answer = toolkit.suite.passAnswer(results, pass.backend, pass.dialect);
+        return { ...answer, requests: requestsOf(toolkit, answers), seconds: seconds(run.started) };
+    });
+}
+/** The default shape's checks, then every shape on its own task, judged against them. */
+async function queueShapes(run) {
+    const { pool, toolkit } = run;
+    const reference = await pool.run({ kind: "shape-reference" });
+    const answers = await Promise.all(toolkit.shapes.SHAPE_NAMES.filter((name) => run.matrix.shapes?.includes(name) ?? true).map((name) => pool.run({ kind: "shape", name, reference: reference.checks })));
+    for (const answer of [reference, ...answers])
+        run.peakRss = Math.max(run.peakRss, answer.memory.rss);
+    return { cases: answers.flatMap((answer) => answer.cases), requests: requestsOf(toolkit, [reference, ...answers]) };
+}
+/**
+ * Recall behind each shape that runs clean, a chunk of defects per task; a shape that does not
+ * waits on itself. Each shape says on stderr when its recall is done, so a long step shows it moves.
+ */
+async function queueRecall(run, shapes) {
+    const { pool, toolkit, writer } = run;
+    const perShape = (await shapes).cases.map(async (shape) => {
+        if (shape.problems.length > 0)
+            return { cases: [toolkit.shapes.waitingRecall(shape)], requests: toolkit.suite.sumRequests([]) };
+        const name = shape.label.replace(/^shape:/, "");
+        const began = performance.now();
+        const answers = await Promise.all(chunked(toolkit.shapes.recallDefects("memory", name)).map((defects) => pool.run({ defects, kind: "recall", shape: name })));
+        for (const answer of answers)
+            run.peakRss = Math.max(run.peakRss, answer.memory.rss);
+        const cases = answers.flatMap((answer) => answer.cases);
+        const unexpected = toolkit.shapes.renderShapeSuite(cases, true).failures;
+        writer.err(`  recall behind ${name}: ${cases.length} case(s), ${unexpected} not as expected · ${seconds(began)}s\n`);
+        return { cases, requests: requestsOf(toolkit, answers) };
+    });
+    const tallies = await Promise.all(perShape);
+    return { cases: tallies.flatMap((tally) => tally.cases), requests: requestsOf(toolkit, tallies) };
+}
+/** The default pass: memory on the default dialect, whose runs the tag-unlock and `--ops` claims are judged from. */
+function defaultPass(matrix) {
+    return matrix.passes.findIndex((pass) => pass.backend === "memory" && pass.dialect === "postgrest");
+}
+/**
+ * Shapes and recall — whether the API tester stays quiet on a correct backend shaped unlike the one it was
+ * written against, and still sees each defect behind it; one engine is enough, a shape varies the
+ * API, not the storage — and `--ops` recall, which needs each defect's full run from the default pass.
+ */
+function queueExtras(run, answers) {
+    const { matrix, pool, toolkit } = run;
+    const shapes = matrix.legs.has("shapes") || matrix.legs.has("recall") ? queueShapes(run) : undefined;
+    const recall = shapes !== undefined && matrix.legs.has("recall") ? queueRecall(run, shapes) : undefined;
+    const index = defaultPass(matrix);
+    const scope = !matrix.legs.has("scope") || index === -1
+        ? undefined
+        : answers[index].then((answer) => {
+            const names = Object.keys(toolkit.defects.DEFECTS);
+            const lanes = Array.from({ length: matrix.jobs }, (_, lane) => names.filter((_name, at) => at % matrix.jobs === lane));
+            return Promise.all(lanes
+                .filter((lane) => lane.length > 0)
+                .map((defects) => pool.run({ defects, kind: "scope", known: answer.primaryOps })));
+        });
+    return { recall, scope, shapes };
+}
+/** The legs asked for, printed in order as each is answered. Returns what failed and what was proven. */
+export async function runMatrix(writer, toolkit, matrix) {
+    const pool = toolkit.pool.createPool(matrix.jobs);
+    const run = {
+        failures: 0,
+        matrix,
+        peakRss: 0,
+        pool,
+        proven: new Set(),
+        started: performance.now(),
+        toolkit,
+        writer,
+    };
+    try {
+        const answers = queuePasses(run);
+        const extras = queueExtras(run, answers);
+        /* Observed from the start: when one leg fails, the rest must not surface as unhandled. */
+        void Promise.allSettled([...answers, extras.shapes, extras.recall, extras.scope]);
+        for (const [index, pass] of matrix.passes.entries()) {
+            writer.out(`\n  ── ${pass.backend} · ${pass.dialect} ${"─".repeat(46)}\n`);
+            const answer = await answers[index];
+            writer.out(`${answer.text}${toolkit.suite.requestLines(answer.requests)}  done in ${answer.seconds}s\n`);
+            run.failures += answer.failures;
+            for (const id of answer.proven)
+                run.proven.add(id);
+        }
+        await judgeWhole(run, answers, extras);
+    }
+    finally {
+        await pool.close();
+    }
+    if (run.peakRss > 0)
+        writer.out(`  largest worker: ${Math.round(run.peakRss / 2 ** 20)} MB\n`);
+    return { failures: run.failures, proven: run.proven };
+}
+/** The tag-unlock claims from the default pass's runs, `--ops` recall, and the shapes and recall. */
+async function judgeWhole(run, answers, extras) {
+    const index = defaultPass(run.matrix);
+    if (index !== -1)
+        print(run, await run.toolkit.suite.runTagUnlockSuite(run.toolkit.suite.listening, (await answers[index]).baselines));
+    if (extras.scope !== undefined) {
+        const scoped = await extras.scope;
+        print(run, scoped.flatMap((answer) => answer.results));
+        run.writer.out(run.toolkit.suite.requestLines(requestsOf(run.toolkit, scoped)));
+    }
+    if (extras.shapes !== undefined)
+        await judgeShapes(run, extras.shapes);
+    if (extras.recall !== undefined)
+        await judgeRecall(run, extras.recall);
+}
+function print(run, cases) {
+    const rendered = run.toolkit.suite.renderParserSuite(cases);
+    run.writer.out(rendered.text);
+    run.failures += rendered.failures;
+}
+async function judgeShapes(run, shapes) {
+    run.writer.out(`\n  ── shapes · memory ${"─".repeat(46)}\n`);
+    const { cases, requests } = await shapes;
+    const rendered = run.toolkit.shapes.renderShapeSuite(cases);
+    run.writer.out(`${rendered.text}${run.toolkit.suite.requestLines(requests)}  done in ${seconds(run.started)}s\n`);
+    run.failures += rendered.failures;
+}
+/** Recall behind the shapes: a defect only a shape can show proves its check there. */
+async function judgeRecall(run, recall) {
+    const { writer, toolkit } = run;
+    writer.out(`\n  ── recall behind shapes · memory ${"─".repeat(32)}\n`);
+    const { cases, requests } = await recall;
+    const rendered = toolkit.shapes.renderShapeSuite(cases, true);
+    writer.out(`${rendered.text}${toolkit.suite.requestLines(requests)}  done in ${seconds(run.started)}s\n`);
+    run.failures += rendered.failures;
+    for (const id of recalledChecks(cases, toolkit.suite.EXPECTED))
+        run.proven.add(id);
+}
+/** The checks clean recall cases prove: each recalled defect's primary check, the first it names. */
+export function recalledChecks(cases, expected) {
+    const proven = [];
+    for (const recalled of cases) {
+        const defect = /^recall:[^:]+:(.+)$/.exec(recalled.label)?.[1];
+        const check = defect === undefined ? undefined : expected[defect];
+        if (check === undefined || recalled.problems.length > 0)
+            continue;
+        proven.push(...[check].flat().slice(0, 1));
+    }
+    return proven;
+}
+//# sourceMappingURL=conformance-matrix.js.map

package/dist/api/cli/commands/conformance-proven.d.ts

@@ -0,0 +1,15 @@
+import type { Writer } from "../writer.ts";
+import type { Toolkit } from "./conformance-toolkit.ts";
+/** What one leg proved, as `--proven-out` writes it. */
+export interface ProvenFile {
+    legs: string[];
+    proven: string[];
+}
+/** Writes the checks a leg proved to `path`, making its folder. */
+export declare function writeProven(path: string, legs: readonly string[], proven: ReadonlySet<string>): void;
+/**
+ * `conformance --aggregate <dir>`: every leg's proven checks together, and the checks none of
+ * them proved. Exits 1 on any, 2 when there is nothing to read.
+ */
+export declare function aggregate(writer: Writer, toolkit: Toolkit, dir: string): number;
+//# sourceMappingURL=conformance-proven.d.ts.map

package/dist/api/cli/commands/conformance-proven.js

@@ -0,0 +1,44 @@
+import { mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { messageOf } from "../../errors/message.js";
+import { EXIT } from "../../runtime/exit.js";
+/** Writes the checks a leg proved to `path`, making its folder. */
+export function writeProven(path, legs, proven) {
+    mkdirSync(dirname(path), { recursive: true });
+    const body = { legs: [...legs], proven: [...proven].toSorted() };
+    writeFileSync(path, `${JSON.stringify(body, null, "\t")}\n`);
+}
+/**
+ * `conformance --aggregate <dir>`: every leg's proven checks together, and the checks none of
+ * them proved. Exits 1 on any, 2 when there is nothing to read.
+ */
+export function aggregate(writer, toolkit, dir) {
+    let files;
+    try {
+        files = readdirSync(dir)
+            .filter((name) => name.endsWith(".json"))
+            .toSorted();
+    }
+    catch (error) {
+        writer.err(`testing api: cannot read ${dir}: ${messageOf(error)}\n`);
+        return EXIT.usage;
+    }
+    if (files.length === 0) {
+        writer.err(`testing api: ${dir} holds no proven-check files; write them with --proven-out\n`);
+        return EXIT.usage;
+    }
+    const proven = new Set();
+    for (const name of files) {
+        const read = JSON.parse(readFileSync(join(dir, name), "utf8"));
+        if (!Array.isArray(read.proven))
+            continue;
+        for (const id of read.proven)
+            if (typeof id === "string")
+                proven.add(id);
+    }
+    writer.out(`  ${files.length} leg file(s) · ${proven.size} check(s) proven\n`);
+    const unproven = toolkit.suite.renderUnproven(proven);
+    writer.out(unproven.text);
+    return unproven.failures > 0 ? 1 : 0;
+}
+//# sourceMappingURL=conformance-proven.js.map

package/dist/api/cli/commands/conformance-suites.d.ts

@@ -0,0 +1,11 @@
+import type { Writer } from "../writer.ts";
+import type { Toolkit } from "./conformance-toolkit.ts";
+/**
+ * The suites that need no defect matrix: the document parser, the documented example, the coverage
+ * report, tenant scope, feature gates, rate limits, uniqueness, the seed contract, effects and the
+ * payload catalog. The tag-unlock claims are judged from the matrix's own runs; only a run that
+ * stops before the matrix makes them here. Every suite that runs the API tester reaches its target through one
+ * counter, and the leg ends with the requests they sent, by source. Returns the failures.
+ */
+export declare function runSuites(writer: Writer, toolkit: Toolkit, beforeMatrix: boolean): Promise<number>;
+//# sourceMappingURL=conformance-suites.d.ts.map

package/dist/api/cli/commands/conformance-suites.js

@@ -0,0 +1,35 @@
+/**
+ * The suites that need no defect matrix: the document parser, the documented example, the coverage
+ * report, tenant scope, feature gates, rate limits, uniqueness, the seed contract, effects and the
+ * payload catalog. The tag-unlock claims are judged from the matrix's own runs; only a run that
+ * stops before the matrix makes them here. Every suite that runs the API tester reaches its target through one
+ * counter, and the leg ends with the requests they sent, by source. Returns the failures.
+ */
+export async function runSuites(writer, toolkit, beforeMatrix) {
+    const { suite } = toolkit;
+    let failures = 0;
+    const print = (cases) => {
+        const rendered = suite.renderParserSuite(cases);
+        writer.out(rendered.text);
+        failures += rendered.failures;
+    };
+    const counted = suite.counting(suite.listening);
+    const { via } = counted;
+    print(suite.runParserSuite());
+    /* The documented example is checked in the same breath: it is the only place a reader sees
+     * where each tag goes, and documentation that drifts teaches a shape that no longer works. */
+    print(await suite.runExampleSpecSuite());
+    if (beforeMatrix)
+        print(await suite.runTagUnlockSuite(via));
+    print(suite.runCoverageReportSuite());
+    print(await suite.runTenantScopeSuite(via));
+    print(await toolkit.featureGate.runFeatureGateSuite(via));
+    print(await toolkit.rateLimit.runRateLimitSuite(via));
+    print(await toolkit.unique.runUniqueSuite(via));
+    print(await toolkit.seedContract.runSeedContractSuite(via));
+    print(await toolkit.effects.runEffectsSuite(via));
+    print(suite.runPayloadCatalogSuite());
+    writer.out(suite.requestLines(counted.requests()));
+    return failures;
+}
+//# sourceMappingURL=conformance-suites.js.map

package/dist/api/cli/commands/conformance-tail.d.ts

@@ -0,0 +1,27 @@
+import { type Args } from "../flags.ts";
+import type { Writer } from "../writer.ts";
+import type { BackendName } from "./conformance-matrix.ts";
+import type { Toolkit } from "./conformance-toolkit.ts";
+/** A local engine a full run went without; `noDriver` when it was Postgres and this install lacks its driver. */
+export interface SkippedBackend {
+    backend: BackendName;
+    noDriver?: boolean;
+}
+/**
+ * An engine that is not there is not a pass: the SQL-only defects and every engine difference go
+ * untested. Leaving one out has to be asked for by name. Returns the failures.
+ */
+export declare function judgeSkipped(writer: Writer, flags: Args["flags"], skipped: readonly SkippedBackend[]): number;
+/**
+ * A short combination pass, part of the standard self-test. Single-defect recall says nothing
+ * about how the diagnosis holds up when several faults overlap, and every bug the fuzzer has found
+ * so far — silent bail-outs, non-transitive suppression, a probe writing into a constrained field —
+ * was invisible to the one-at-a-time matrix. A fixed seed tests the same 40 combinations forever;
+ * CI passes a fresh one per run, and the seed is printed so any failure replays exactly. The flag
+ * parser has already refused a seed that is not a positive integer. The pass ends with the requests
+ * its combinations sent, by source. Returns the failures.
+ */
+export declare function runCombinations(writer: Writer, flags: Args["flags"], toolkit: Toolkit): Promise<number>;
+/** Present but never default: D1 is remote, so a pass costs minutes and someone's quota. */
+export declare function noteD1(writer: Writer, toolkit: Toolkit): void;
+//# sourceMappingURL=conformance-tail.d.ts.map

package/dist/api/cli/commands/conformance-tail.js

@@ -0,0 +1,61 @@
+import { NODE_FLOOR } from "../../errors/node-floor.js";
+import { str } from "../flags.js";
+import { POSTGRES_RANGE } from "../../reference/stores/postgres-driver.js";
+/* Why the engine is missing, and what makes it present. */
+function absence({ backend, noDriver }) {
+    if (backend === "sqlite")
+        return {
+            fix: "Start it",
+            why: `node:sqlite unavailable on Node ${process.version}; the API tester needs ${NODE_FLOOR} or later`,
+        };
+    if (noDriver === true)
+        return { fix: `Run npm install postgres (${POSTGRES_RANGE})`, why: "driver not installed" };
+    return { fix: "Start it", why: "no Postgres server reachable on the default connection" };
+}
+/**
+ * An engine that is not there is not a pass: the SQL-only defects and every engine difference go
+ * untested. Leaving one out has to be asked for by name. Returns the failures.
+ */
+export function judgeSkipped(writer, flags, skipped) {
+    if (skipped.length === 0)
+        return 0;
+    const allowed = new Set(str(flags, "skip-backend")?.split(",") ?? []);
+    let failures = 0;
+    for (const entry of skipped) {
+        const { backend } = entry;
+        const { why, fix } = absence(entry);
+        if (allowed.has(backend)) {
+            writer.out(`  note: ${backend} backend skipped as asked — ${why}\n`);
+            continue;
+        }
+        failures += 1;
+        writer.out(`  ✗ ${backend} backend unavailable — ${why}. ${fix}, or pass --skip-backend ${backend}\n`);
+    }
+    writer.out("\n");
+    return failures;
+}
+/**
+ * A short combination pass, part of the standard self-test. Single-defect recall says nothing
+ * about how the diagnosis holds up when several faults overlap, and every bug the fuzzer has found
+ * so far — silent bail-outs, non-transitive suppression, a probe writing into a constrained field —
+ * was invisible to the one-at-a-time matrix. A fixed seed tests the same 40 combinations forever;
+ * CI passes a fresh one per run, and the seed is printed so any failure replays exactly. The flag
+ * parser has already refused a seed that is not a positive integer. The pass ends with the requests
+ * its combinations sent, by source. Returns the failures.
+ */
+export async function runCombinations(writer, flags, toolkit) {
+    writer.out(`\n  ── combinations ${"─".repeat(46)}\n`);
+    const seed = Number.parseInt(str(flags, "fuzz-seed") ?? "1", 10);
+    writer.out(`  fuzz seed ${seed} — replay with --fuzz 40 --max-defects 6 --seed ${seed}\n`);
+    const counted = toolkit.suite.counting(toolkit.suite.listening);
+    const smoke = toolkit.fuzz.renderFuzz(await toolkit.fuzz.runFuzz({ backend: "memory", cases: 40, maxDefects: 6, seed, via: counted.via }));
+    writer.out(`${smoke.text}${toolkit.suite.requestLines(counted.requests())}`);
+    return smoke.failures;
+}
+/** Present but never default: D1 is remote, so a pass costs minutes and someone's quota. */
+export function noteD1(writer, toolkit) {
+    writer.out(toolkit.suite.d1Available()
+        ? "  note: d1 backend available — run with --backend d1 (remote, ~2min per defect)\n\n"
+        : "  note: d1 backend needs CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_D1_DATABASE_ID, CLOUDFLARE_API_TOKEN\n\n");
+}
+//# sourceMappingURL=conformance-tail.js.map