@workerstack/testing 0.0.0-stage → 0.31.0

package/docs/api/configuration/principals-and-security.md

@@ -0,0 +1,173 @@
+# testing api — Principals and security
+
+[← back to the overview](../../../README.md)
+
+Who the API tester acts as, and how each OpenAPI security scheme gets its value.
+
+## Principals
+
+```ts
+principals: {
+  alpha: {
+    email: "alpha@example.test",          // identity data: recipes read it as {principal.email}
+    tenant: "acme",                       // principals in different tenants are isolation peers
+    scope: { org_id: "org_alpha" },       // path or query parameters bound for this principal
+    rank: 2,                              // higher can do everything a lower rank can; default 0
+    inviteAs: "key_beta",                 // how an owner names this principal in an invite body
+    readOnly: true,                       // a viewer: its every write must be refused (auth.role-enforced)
+    security: { apiKey: { static: "${KEY_ALPHA}" } }, // this principal's own recipe for a scheme
+  },
+}
+```
+
+The key is the principal's id, the stable name in reports. One principal is the writer: it makes the records every check works on, and the rest act on them. `writer: "<id>"` at the top level names it; unset, the first principal in the map writes. A sort-keys lint rule reorders a map's keys, and with them the writer, so a config under one names its `writer`:
+
+```ts
+writer: "alpha",
+principals: { admin: { … }, alpha: { … }, beta: { … } },
+```
+
+A `writer` that names no principal exits 2, listing the principals there are. `testing api doctor` prints who writes and why. Every other field that is not one of the named ones is identity data, read only as `{principal.<key>}` in recipes; it never fills a parameter.
+
+Rules that matter:
+
+- **Isolation** (`tenant.*`) needs two principals whose `tenant` differs.
+- **Rank** (`auth.rank-is-monotonic`) needs two principals in the _same_ tenant with different `rank`.
+- **Invite** (`auth.invite-grants-then-revokes`) needs `x-invite` on the spec and a _different-tenant_ principal with `inviteAs` set.
+- Extra principals are not ignored. Isolation picks the first different-tenant peer. Rank uses the same-tenant pair.
+- `readOnly: true` marks an account that may only read. The security checks send its writes to a record the run created and expect each to be refused. When present it must be `true`.
+- `scope` is the only way a principal binds a parameter. A login can bind `tenant` and `scope.<param>` from its answer with `claims`.
+- `security.<scheme>` overrides the base recipe for this principal alone: one naming a kind replaces it (keeping `format` and `placements`); one naming only options merges over it.
+
+Every account the run makes is ledgered before its request goes out, whatever sends it: a `fresh: true` principal, a disposable session, a login recipe whose step calls the operation the document marks `x-fresh-principal`, a scenario, or a check's generated body sent to that operation or to a create of the `identity: self` entity. A principal with `created: true` counts too: it is the opt-in for a login that registers through an operation the document does not mark. The ledger is `accounts.json` in the run folder. It keeps the names each account is found by (email, username, login, handle, phone), never its password, so a run that dies still names what it made.
+
+At the end of a run, each account is removed in this order, each tried only when the one before could not:
+
+1. The document's delete-self (a `DELETE` tagged `x-entity: { identity: self, action: delete }`), sent as the account itself.
+2. The delete of the entity the sign-up creates, by the account's id, as each configured principal the run did not make, strongest first.
+3. `hooks.teardownPrincipal`, with the account's address and the last credential the run holds for it.
+
+A sign-up the document leaves untagged counts too. That is an open POST (its security is `[]`, or every alternative is empty) whose operationId or last path segment ends with the words register, registration, signup or sign up, as `/auth/admin-sign-up` does. It is also an open collection create whose body takes a `password`, a `passphrase` or a `format: password` property and that is named for no login, token or session, nor for a password reset, forgotten, recovered or changed. The API tester ledgers what it sends there like any sign-up. With no tag, it has no delete-self to send and cannot sign in as the account, so such an account is usually reported as a leftover: tag the operation `x-fresh-principal` to let the API tester sign in and remove it.
+
+An account none of these removes is a leftover and fails the run with exit 4, named by its email or username so a person can remove it. An account whose name a configured principal holds, which the run did not make, is never removed, whatever answered its sign-up, so a long-lived test account survives every run. Teardown never sends a sign-up again to get a credential. Generated values for the fields that name an account carry the run's nonce, so two runs never send the same email.
+
+Example — two tenants plus a same-tenant lattice:
+
+```ts
+security: {
+	bearer: { login: { steps: [{ operationId: "auth.token", body: { key: "{principal.key}" } }], from: "$.access_token" } },
+},
+principals: {
+	alpha: { key: "key_alpha", tenant: "org_alpha", scope: { org_id: "org_alpha" }, rank: 2 },
+	alpha_member: { key: "key_alpha_member", tenant: "org_alpha", scope: { org_id: "org_alpha" }, rank: 1 },
+	beta: { key: "key_beta", tenant: "org_beta", scope: { org_id: "org_beta" }, rank: 2, inviteAs: "key_beta" },
+},
+```
+
+## Security recipes
+
+Each operation is sent the credentials its OpenAPI `security` requires, placed where its scheme says: a bearer token in `Authorization`, an apiKey in its header, query parameter or cookie. `security` says only how to get each scheme's value, one recipe per scheme the document declares:
+
+```ts
+security: {
+	jwt: {
+		login: {
+			steps: [{ operationId: "auth.login", body: { email: "{principal.email}", password: "{principal.password}" } }],
+			from: "$.access_token",
+			claims: { tenant: "org_id", "scope.project_id": "$.project.id" },
+		},
+		refresh: { steps: [{ operationId: "auth.refresh", body: { refresh_token: "{refreshToken}" } }], from: "$.access_token" },
+		expiresIn: { from: "$.expires_in" },
+	},
+	apiKey: { static: "${API_KEY}", format: "ApiKey {credential}" },
+},
+```
+
+A recipe has exactly one kind:
+
+| kind                                                   | the value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `login: { steps, from, claims?, revoke? }`             | Run the steps; `from` is where the credential is in the last answer. `claims` binds `tenant` and `scope.<param>`: a value starting with `$` is a JSON path into the last answer's body, any other a JWT claim name. `revoke` names the operation that ends a session, alone or as `{ operationId, body }` with `{credential}` in the body: the API tester ends the sessions its probes open with it, and each principal's own at the very end, unless `--keep-credentials`. It is sent with the session it ends, so a 401 counts as revoked. Name every operation that ends sessions here (a logout, a session or token delete): the contract floor then calls it with a dedicated credential, never a principal's own session |
+| `mint: { with, operationId, body?, from, revoke? }`    | Call `operationId` with the `with` scheme's credential; `{scopes}` in `body` becomes the scopes an operation needs. `revoke` runs at the end                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+| `static: "…"` or `{ username, password }`              | A fixed value; a basic scheme takes the pair                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
+| `hook: true`                                           | `hooks.resolveCredential` returns it. See [Credentials from a hook](#credentials-from-a-hook)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `oauth2: { flow, clientId, clientSecret?, scopes? … }` | A `clientCredentials` or `password` grant at the `tokenUrl` the document declares                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+
+Options on any recipe: `format` replaces the scheme's value template (`ApiKey {credential}` for an apiKey sent in `Authorization`); `expiresIn` is seconds or `{ from }` in the issuing answer, else a JWT `exp`; `refresh` renews a login by its own steps; `holds` lists the scopes a static or hooked credential carries; `placements` adds places: `[{ in: "header" | "query" | "cookie", name, template? }]`.
+
+An operation with alternatives is sent the first one, in document order, whose every scheme the principal has a recipe for. `security: []` or `{}` sends nothing. A 401 on a request sent with a stored credential renews it once and retries once; a second 401 is evidence. A declared `refresh` that fails falls back to one fresh login, never one that signs up again: not a login that calls the `x-fresh-principal` operation, and not any login of a `created: true` principal, whose login registers its account. A credential the run adopts, such as the one an invite accept issues, renews by the declared refresh from what it came with; when that refresh reads a value it did not come with (a `{refresh_token}`), it renews the principal's own session instead. A login that fails blocks that principal for that scheme for the rest of the run: the refused request keeps its 401, what needs the credential later is untested with the reason, and it is not retried.
+
+The config is checked against the document and each origin's before anything is sent; every mistake is named, with the fix, and the CLI exits 2. That covers a recipe for a scheme the document lacks (with the nearest name), two kinds or none, a mint that loops or names no operation, a `{principal.<key>}` a principal does not have, and a `globalHeaders` entry where a credential goes.
+
+## Login steps
+
+Each step is one of:
+
+**Operation step** (prefer this — survives the path moving):
+
+```ts
+{
+  operationId: "auth.token",
+  body: { key: "${API_KEY}" },
+  headers: { "x-extra": "1" },
+  query: { realm: "test" },
+  saveAs: { credential: "$.access_token", refreshToken: "$.refresh_token" },
+  saveClaimsFrom: { token: "$.access_token", bind: { orgId: "orgs.0.oid" } },
+  bind: { address: "user@example.test" }, // literals, with {name} interpolation
+  expect: [200],                            // default: any 2xx
+}
+```
+
+**Request step** (when the document has no sign-in operation, or the hop is not in the spec):
+
+```ts
+{
+  method: "POST",
+  path: "/v1/auth/register/email",
+  body: { email: "{address}", password: "…" },
+  bind: { address: "testing-alpha@example.test" },
+}
+```
+
+`path` is joined to this origin's `baseUrl`, unless it is an absolute `http://` or `https://` URL — then the API tester dispatches to that URL as a recorded exchange (method, URL, status, redirects, response headers). That is how a consume page on the app origin is a login step when `baseUrl` is the API. Unknown host is allowed. `redirect` defaults to `"follow"` with a per-request cookie jar, so a 303 that sets `Set-Cookie` still leaves that cookie visible to `saveAs`. `redirect: "manual"` stops at the first 3xx (`expect: [303]` then bind from that response). An `operationId` step, and every call the API tester makes on its own, stops at the first 3xx when that operation documents a 3xx: the redirect is the documented answer, and the page behind it is not. A required query, header or cookie parameter the request leaves unnamed is sent with the value the document declares for it, written in its declared style ([Parameters](../model.md#parameters)); with no declared value it is left out, never invented. Naming it, even as `undefined`, keeps it the caller's.
+
+**Out-of-band step** (email link, OTP — the API tester cannot collect this itself):
+
+```ts
+{ outOfBand: { address: "{address}", kind: "email-verify", as: "verifyLink" } }
+```
+
+The hook returns a string. If that string is a URL, a later RequestStep must GET it — a hook-side `fetch` is not a recorded exchange.
+
+The API tester tells the hook when the request this value answers was sent (`since`): the latest earlier request in the flow or scenario that names `address` in its URL or body. A later request to someone else does not move it. When no request names the address, the API tester passes no `since`. A value sent before the scenario began — by a scheduled job, or into a seeded inbox — sets `sentEarlier: true`, so a sign-in that names the address does not hide it:
+
+```ts
+{ outOfBand: { address: "{offerEmail}", as: "offerToken", kind: "upsell-offer", sentEarlier: true } }
+```
+
+Later steps interpolate `{name}` from the flow scope, `saveAs` addresses included (`cookie:rt.{userId}` reads a per-user cookie). A name is letters, digits and `_`, with single hyphens between them, as path parameters are written: `{user-id}`. A `{name}` still unfilled when a step is sent, in its path, body, headers or query, fails the step as the API tester's own error naming the placeholder; it never reaches the backend. `saveAs` addresses:
+
+- `$.foo.bar` / `$.orgs.0.id` — JSON body (dot + numeric index only; no JSON Pointer, no filters)
+- `cookie:<name>` — that cookie on `Set-Cookie` for this hop, including followed hops
+- `header:<name>` — a non-set-cookie response header, case-insensitive
+
+A login or refresh `from` and its steps' `saveAs`, an `oauth2` grant's tokens, and `expiresIn: { from }` read an answer whose media type is `application/x-www-form-urlencoded` (parameters such as `; charset=utf-8` ignored) as a form: `access_token` or `$.access_token` names a field. Form answers support top-level keys only. Any other answer is read as JSON, never sniffed, and a value it does not hold fails with a reason that names its type. Every value read from a form is a secret before the answer is recorded, as from JSON.
+
+Missing cookie / header / JSON path fails the step closed. Bind `saveAs.refreshToken` from `$.refresh_token` or `cookie:refresh` so `{refreshToken}` interpolates in `refresh`. A `{name}` in a login or refresh step that nothing fills (an earlier `saveAs`, `bind`, `saveClaimsFrom` or out-of-band `as`; for a refresh, also what the login saved and `{credential}`) is a config error naming it; where only the run knows the value, a signup's answer say, the step is refused before it is sent and the principal is BLOCKED for that scheme. `saveClaimsFrom.token` is a JSON path into the response body (`$.access_token`) or a scope key already bound by `saveAs` (`credential`). Signature is not verified — the API tester is reading its own credential.
+
+`bind` on a step runs **before** the request. `saveAs` then `saveClaimsFrom` run **after**. `from` is read from the last HTTP response unless a step already saved `credential`. `outOfBand.as = "credential"` still wins over `from`.
+
+A login that registers the account it signs in to is torn down only when it calls the `x-fresh-principal` operation or the principal sets `created: true`; see [Principals](#principals).
+
+## Credentials from a hook
+
+When the credential is produced outside the API tester (a human finishes Google OAuth on a harvest page, a pair lands in KV, a TOTP code), do not make the API tester drive `authorize` / `callback`. Declare a hook recipe:
+
+```ts
+security: { google: { hook: true } },
+hooks: {
+	resolveCredential: async (scheme, principal) => readHarvested(scheme, principal.id),
+},
+```
+
+The API tester calls `hooks.resolveCredential(scheme, { id, identity })` on every acquire, so a renewal calls it again. Return the value, `{ value, expiresIn? }`, or `null`, which blocks that principal for that scheme with the reason in the report. The API tester does not speak OAuth.

package/docs/api/configuration/reference.md

@@ -0,0 +1,283 @@
+# testing api — Configuration reference
+
+[← back to the overview](../../../README.md)
+
+The config file, field by field. The rest of the configuration docs: [principals and security](principals-and-security.md), [hooks](hooks.md), [origins and uploads](origins-and-uploads.md); upgrading from 0.24: [migrating to 0.25](../migrating-0.25.md).
+
+## Complete configs
+
+These are copy-paste starting points. Real configs import `defineConfig` from `@workerstack/testing`. Files inside this repository import from `../dist/index.js` because they live in the source tree.
+
+### Starting from testing api init
+
+`testing api init --spec <url|file>` writes a starter `testing.api.config.ts` from the document: one `security` recipe per scheme it declares, each secret read with `env()`, and one principal. Set the env vars it prints, then run `testing api doctor --config testing.api.config.ts`. A document that declares no scheme gets a commented `overlay.securitySchemes` stub to declare one. See [`testing api init`](../commands.md#testing-api-init).
+
+### Static API keys, two tenants (smallest useful)
+
+```ts
+// testing.api.config.ts
+import { defineConfig } from "@workerstack/testing/api"
+
+export default defineConfig({
+	spec: "https://api.example.com/openapi.json",
+	baseUrl: "https://api.example.com",
+	security: { bearer: { static: "${API_TOKEN}" } },
+	principals: {
+		alpha: { tenant: "a", scope: { project_id: "${PROJECT_A}" } },
+		beta: { tenant: "b", scope: { project_id: "${PROJECT_B}" }, security: { bearer: { static: "${API_TOKEN_B}" } } },
+	},
+})
+```
+
+```bash
+export API_TOKEN=… API_TOKEN_B=… PROJECT_A=proj_a PROJECT_B=proj_b
+testing api run --config testing.api.config.ts
+```
+
+`bearer` names a scheme in the document's `components.securitySchemes`; the API tester places the value where that scheme says. One principal is enough to seed and run CRUD / query / schema checks. The second principal, in a different `tenant`, is what makes `tenant.*` run. Without it those checks are **did not apply**, not a pass.
+
+Shipped as `labs/api/minimal.config.ts` (and in the npm package).
+
+### JSON config
+
+Same object. `${NAME}` is interpolated after load. There is no `defineConfig` wrapper.
+
+```json
+{
+	"spec": "https://api.example.com/openapi.json",
+	"baseUrl": "https://api.example.com",
+	"security": { "apiKey": { "static": "${API_KEY}", "format": "ApiKey {credential}" } },
+	"principals": { "alpha": { "tenant": "a", "scope": { "project_id": "${PROJECT_A}" } } },
+	"seed": 42,
+	"cohortSize": 7,
+	"outDir": "./.testing/api/runs"
+}
+```
+
+```bash
+testing api run --config testing.api.config.json
+```
+
+### Demo server (operation-id login)
+
+Shipped as `labs/api/local.config.ts`. Points at `testing api serve`.
+
+```ts
+import { defineConfig } from "@workerstack/testing/api"
+
+export default defineConfig({
+	spec: "/v1/openapi/spec",
+	baseUrl: "http://127.0.0.1:8787",
+	seed: 42,
+	security: {
+		jwt: {
+			login: { steps: [{ operationId: "auth.token", body: { key: "{principal.key}" } }], from: "$.access_token" },
+		},
+	},
+	principals: {
+		alpha: { key: "key_alpha", tenant: "proj_alpha", scope: { project_id: "proj_alpha" }, rank: 2 },
+		beta: { key: "key_beta", tenant: "proj_beta", scope: { project_id: "proj_beta" }, rank: 2 },
+	},
+})
+```
+
+`spec: "/v1/openapi/spec"` is resolved against `baseUrl`. `--base-url` on the CLI overrides the origin without editing the file.
+
+### Two tenants plus a same-tenant rank lattice
+
+See [Principals](principals-and-security.md#principals). Isolation keys off `tenant`. Rank keys off `rank` within one tenant.
+
+## Config files
+
+Two inputs, always: **the spec** and **a config file**. Backend-specific knowledge lives in `x-*` tags and this file. A backend adopts the API tester by adding tags, not by adapting to the API tester.
+
+A config is:
+
+- `.ts` / `.js` / `.mjs` with `export default defineConfig({ ... })`, or
+- `.json` with the same object.
+
+Named export without `default` is also accepted (`module.default ?? module`).
+
+## Top-level fields
+
+```ts
+import { defineConfig } from "@workerstack/testing/api"
+
+export default defineConfig({
+	spec: "https://api.example.com/openapi.json", // URL or filesystem path; JSON or YAML
+	baseUrl: "https://api.example.com",
+	security: { bearer: { static: "${API_TOKEN}" } }, // one recipe per scheme the document declares
+	principals: { alpha: { tenant: "acme" } }, // at least one; see below
+	hooks: {/* optional */},
+	uploads: { pool: ["./fixtures/**/*"], eachMax: 24 },
+	globalHeaders: { "x-request-id": "the API tester" }, // sent on every request; never a credential
+	redact: { headers: ["x-tester-key"] }, // secret, but not a credential
+	seed: 42, // fixture generation; a failing run with the same seed is identical
+	cohortSize: 12, // records created per entity (default 7)
+	maxInFlight: 4, // HTTP in flight
+	ops: ["store.create", "product.*"], // grade only these operationIds; the rest is support
+	only: ["store", "product"], // grade every operation these entities own (union with ops)
+	keepFixtures: false,
+	outDir: "./.testing/api/runs",
+	saveExchanges: true, // default on unless --profile cheap; --quiet does not turn this off
+	network: { retries: 4, waitMs: 60_000 }, // fetch-threw: retry, then wait for the link
+	outOfBand: { attempts: 20, initialMs: 1000, maxMs: 8000 },
+	origins: [
+		{
+			id: "cdn",
+			baseUrl: "https://cdn.example.com",
+			spec: "https://cdn.example.com/openapi.json",
+			security: { bearer: "bearer" },
+		},
+	],
+	query: {
+		// grammar / sortGrammar: a preset name or a template, as on x-query; outranks the document
+		grammar: "postgrest",
+		// the parameter each role is sent in, where no alias finds it; outranks x-query.params
+		params: { filter: "filter_by" },
+		// the response field each paging fact is read from, where no alias finds it; outranks x-query.envelope
+		envelope: { nextCursor: "next_cursor_id" },
+		operators: ["eq", "neq", "gt", "gte", "lt", "lte", "in", "nin", "like", "ilike", "is"],
+		emptyIn: "match-none",
+		maxInValues: 100,
+		searchEmpty: "match-all",
+		sort: { nulls: ["first", "last"], maxKeys: 3 },
+		select: { unknown: "reject" },
+	},
+	entities: {
+		row: {
+			query: {
+				// an entity's grammar outranks the run's
+				sortGrammar: { term: "{field}_{direction}", asc: "ascending", desc: "descending", join: "," }, // join: "repeat" sends one sort parameter per key
+				identityFilter: "_id",
+				filterable: [{ field: "_id", type: "string", ops: ["eq", "neq", "in"] }],
+			},
+		},
+	},
+})
+```
+
+| field           | required | default                                                                           | notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| --------------- | -------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `spec`          | yes      |                                                                                   | See [Spec loading](#spec-loading)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `baseUrl`       | yes      |                                                                                   | Primary origin. OpenAPI `servers[]` is ignored                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| `principals`    | yes      |                                                                                   | A map of id → identity data, at least one. The writer is the one `writer` names, or the first. See [Principals](principals-and-security.md#principals)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `writer`        | no       | the first principal                                                               | The principal that writes: it makes the records every check works on. A name that is no principal is an error that lists the principals                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `security`      | no       |                                                                                   | One recipe per scheme the document declares: how each credential is obtained. See [Security recipes](principals-and-security.md#security-recipes)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `hooks`         | no       |                                                                                   | See [Hooks](hooks.md#hooks)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `uploads`       | no       |                                                                                   | `pool` globs; optional `each` (operationId → globs) and `eachMax`. JSON configs may set all three                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `globalHeaders` | no       | `{}`                                                                              | Merged first, then `resolveHeaders`, then caller headers. Never a credential: a credential header, or one where a scheme places its credential, exits 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `cache`         | no       | `{ bypass: true }`                                                                | Every documented read is sent the cache bypass its `x-consistency` declares, else `Cache-Control: no-cache`, so a write is verified against the database. `bypass: false` sends none (a firewall refuses the header, or the caches are under test). A header `globalHeaders` or `resolveHeaders` sets under that name wins                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| `redact`        | no       |                                                                                   | `{ headers: ["x-tester-key"] }`: header names whose values are secret though they carry no credential, scrubbed wherever a run records them                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `origins`       | no       | `[]`                                                                              | Extra `{ id, baseUrl, spec, maxRps?, overlay?, security? }` hosts. `security` maps an origin scheme to a primary recipe. Do not merge those routes into `spec`. See [Secondary origins](origins-and-uploads.md#secondary-origins)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| `outOfBand`     | no       | `{ attempts: 6, initialMs: 200, maxMs: 3000 }`                                    | Backoff for `resolveOutOfBand`. See [Hooks](hooks.md#hooks)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `seed`          | no       | `1`                                                                               | Integer. Same seed → same fixture bodies                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| `cohortSize`    | no       | `7`                                                                               | Sliced from the 7 built-in variants. Larger repeats the pattern                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| `maxInFlight`   | no       | `4`                                                                               | Across the whole run                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| `maxRps`        | no       | none                                                                              | Requests per second to `baseUrl` in all, across every route: a limit no route declares, such as a CDN or firewall capping each client IP. Route budgets (`rateLimits`, `x-rate-limit`) still hold under it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| `checks`        | no       | none on                                                                           | Checks that run only when switched on, by id: `{ "auth.login-throttled": true }` sends failed logins for an identity that does not exist until the login answers 429. An unknown id is a config problem                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| `payloads`      | no       | `"per-write-path"`                                                                | String payload catalog: all 159 cases once per write path (media type + field type), one per family elsewhere, said in a run note; `"full"` sends every case to every field                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+| `profiles`      | no       | `full` and `cheap`                                                                | Named profiles beyond the two built in: `{ maxCost?, excludeDestructive?, exclude?, excludeExternalKinds? }`. `maxCost` (`low`, `medium` or `high`) excludes an operation whose `x-cost` is higher, `excludeDestructive: true` every `x-destructive` operation, `exclude` the operationIds it lists, and `excludeExternalKinds` every scenario with a step whose external has one of those `externals.kinds`. An entry named `full` or `cheap` replaces that built-in. See [`--profile`](../tags.md#--profile--cost-gating)                                                                                                                                                                                                                                                                                                                        |
+| `profile`       | no       | `"full"`                                                                          | The active profile, by name. `--profile` overrides it. A name neither built in nor in `profiles` is an error that lists the known ones                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `ops`           | no       | every operation                                                                   | The operations to grade, each by its [operation name](#operation-names); every other operation is support, called to reach or observe a target and never graded. `*` globs within a name (`row.*`), and `<originId>:<operationId>` names an operation on a secondary origin. Joins `only` as a union. A name that matches nothing is a usage error (exit 2). `--ops` replaces the list. See [Targeted runs](../commands.md#targeted-runs---ops)                                                                                                                                                                                                                                                                                                                                                                                                    |
+| `only`          | no       | all                                                                               | Entity names from `testing api plan`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| `keepFixtures`  | no       | `false`                                                                           | Skip DELETE at the end                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `overlay`       | no       |                                                                                   | Patches to a document you cannot change: `{ operations: { "<operation name>": { … } }, schemas: { "<Name>": { … } }, securitySchemes: { "<name>": { … } }, security: [{ "<name>": [] }] }`. Objects merge, arrays replace, `null` deletes a key, a patch's own `operationId` renames the operation. Each `operations` key is an [operation name](#operation-names) as the document gives it before any patch. A patch naming nothing is a usage error (exit 2), as are two keys naming one operation (`GET /a` and `get /a`), except `securitySchemes`, whose purpose is to declare a scheme the document lacks (`null` deletes one); `security` replaces the root requirement. Both apply before anything reads the document, so `doctor`, `plan` and `run` all see them. `origins[].overlay` does the same for an origin                         |
+| `allow`         | no       |                                                                                   | The only operations the API tester may call, each by its [operation name](#operation-names). Every other operation leaves the document after the overlay, so no check, contract call, scenario or teardown can reach it; `testing api doctor` lists what was removed. An entry naming nothing, or a recipe's login or refresh step, mint, revoke or scenario step needing a removed operation, is a usage error                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| `fence`         | no       | `{ failRun: true }`                                                               | With `hooks.beforeRequest`: whether a refused request alone fails the run (exit 3)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `baseline`      | no       |                                                                                   | Path, relative to the config, to a file `testing api baseline` writes: `{ "version": 1, "entries": [{ "check", "entity"?, "operationId"?, "reason" }] }`. Matched findings are reported as known and do not fail the run; unmatched entries are reported as fixed. A file the API tester cannot read is a usage error                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| `targetVersion` | no       |                                                                                   | Where the backend says which build it is, read once after sign-in: `{ operationId \| path, from, scheme? }`, with `from` a `$.json.path` into the body or `header:<name>`. An operation is read as the first principal under its security; a `path` is read anonymously, or as the first principal sending `scheme`, a scheme the document declares. Reports and `testing api diff` show it; a version that cannot be read is noted and never changes the verdict                                                                                                                                                                                                                                                                                                                                                                                  |
+| `teardown`      | no       | `{ verify: true }`                                                                | After teardown, read each list again with the creator's credential; a record still listed after its delete answered 2xx is a leftover (`delete answered 2xx but the record is still listed`). `verify: false` skips the read                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `outDir`        | no       | `./.testing/api/runs`                                                             | History root, relative to the config (the default, to the working directory). Each run writes `<outDir>/<datetime>/` and updates `latest`, with `principals.json`, which holds no credential value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `saveExchanges` | no       | on unless `profile` is `cheap`                                                    | Persist every HTTP exchange under the run dir. `--save-exchanges` / `--no-save-exchanges` override. `--quiet` does not                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+| `network`       | no       | `{ retries: 4, waitMs: 60000, requestTimeoutMs: 180000, maxRetryAfterMs: 60000 }` | When `fetch` throws (offline / DNS / reset / timeout): retry, then wait once for the link. `requestTimeoutMs: 0` disables the per-attempt cap. `maxRetryAfterMs` is the longest `Retry-After` a 429 or 503 is waited out for; a longer one is neither waited for nor retried, and its finding names the wait and the limit                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+| `query`         | no       |                                                                                   | Global query-catalog defaults. Overlay after `x-query`. Does not invent operators. `params` names the parameter each role is sent in (`filter`, `sort`, `select`, `search`, `searchMode`, `limit`, `page`, `offset`, `cursor`), run-wide, outranking `x-query.params` on each operation that declares the name. A name no operation of the document or any origin declares, or an unknown role, stops the run. `envelope` names the response field each paging fact is read from (`total`, `hasMore`, `nextCursor`) the same way, outranking `x-query.envelope` on each list whose response declares the field; a field no list response of the document or any origin declares stops the run. To name a parameter or field for one operation, use `overlay.operations.<id>["x-query"].params` or `.envelope`; `entities.<name>.query` has neither |
+| `entities`      | no       |                                                                                   | Per-entity overlays. This release only reads `query`. Unknown names are ignored; `doctor` warns                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| `scenarios`     | no       | `[]`                                                                              | Declared flows for operations the matrix cannot reach. See [Scenarios](#scenarios)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+| `externals`     | no       |                                                                                   | `{ kinds: { <external>: <kind> } }`: each scenario external's kind, in your own words (`externalKinds` in [`@workerstack/testing/api/steps`](../steps.md#externals) builds it from a registry). A profile's `excludeExternalKinds` leaves out a scenario that needs an excluded kind: nothing is sent for it, and it is a `profile.skip` gap naming the external, its kind and the profile, so what it covers stays ungraded                                                                                                                                                                                                                                                                                                                                                                                                                       |
+
+`spec` may be a path relative to `baseUrl` (`/v1/openapi/spec`) or an absolute URL or a file. Every relative file path a config holds (`spec`, each origin's `spec`, `baseline`, `outDir`, upload patterns) is read from the config file's folder, whatever the working directory.
+
+CLI `--base-url`, `--ops`, `--only`, `--seed`, `--out`, `--max-in-flight`, `--keep-fixtures`, `--save-exchanges` / `--no-save-exchanges` override these when passed.
+
+## Operation names
+
+`ops`, `allow` and `overlay.operations` name an operation the same way, through one resolver. An
+operation is named by its `operationId`, or, when the document gives it none, `METHOD /path`, as
+in `"GET /admin/realms/{realm}/users"`. The method may be written in any case; the path must be
+one of the document's path templates exactly, so `/users/` and `/users/{id}` for `/users/{userId}`
+match nothing. An operation with an `operationId` answers to that id alone, the name every report
+prints. A name that matches no operation is a usage error that suggests the nearest names.
+
+## Spec loading
+
+Resolved in this order, never by guessing the string's "look":
+
+1. Absolute `http(s)://` or `file://` — used as given.
+2. A path that exists on disk, relative to the config file's folder (a `--spec` on the command line: the working directory).
+3. Anything else, when `baseUrl` is known — resolved against it (`/v1/openapi/spec`, `openapi.json`).
+
+JSON if the first non-space character is `{` or `[`, otherwise YAML. Empty files error. A JSON document with more opening than closing brackets is diagnosed as truncated (proxy / download limit), not as a syntax error.
+
+The API tester reads OpenAPI 3.0–3.2 only. The API tester reads `paths`, operations, parameters, request/response JSON schemas, and `x-*` extensions. A Swagger 2.0 document, an OpenAPI version below 3.0, or one newer than 3.2 is refused with exit 2, naming its version (`testing api: api.json is Swagger 2.0; the API tester reads OpenAPI 3.0–3.2 only — convert it first`). A document with no `openapi` version string is not an OpenAPI document and fails to load.
+
+Internal `$ref`s are dereferenced. When the document is a file, a `$ref` to another file by relative path (`./schemas/pet.yaml#/Pet`) is read too, relative to the file that names it, recursively, so a document split across files loads as one; a cycle across files works like one inside a file. Every such file must resolve inside the document's own folder, compared by real path: a ref by absolute path or `file:` URL, one that climbs out with `..`, or a symlink that points outside is refused with exit 2, as is a ref to a missing file (the error names the file and the file that refers to it). URL refs, anchor refs (`#Name`), and every ref in a document fetched by URL stay unresolved and show up in `testing api doctor` / the JSON `externalRefs` list.
+
+## Scenarios
+
+Some operations need a scene the document cannot describe: a second factor enabled before it can be verified, a checkout paid before the webhook lands, an invite accepted by someone other than the inviter. A scenario declares that scene as steps. The API tester runs them and grades each operation in `covers` by what the steps observed. The product knowledge lives in your config; the API tester stays generic.
+
+```ts
+scenarios: [
+	{
+		id: "rename",
+		covers: ["notes.create", "notes.rename"],
+		steps: [
+			{ operationId: "notes.create", body: { text: "first" }, saveAs: { id: "$.id" } },
+			{ operationId: "notes.list", assert: { from: "$.items[*].text", contains: "first", observes: ["notes.create"] } },
+			{ operationId: "notes.rename", bind: { id: "{id}" }, body: { text: "second" } },
+			{ operationId: "notes.list", assert: { from: "$.items[*].text", contains: "second", observes: ["notes.rename"] } },
+		],
+	},
+],
+```
+
+Steps are the login steps (`operationId` / raw `method` + `path`, `body`, `bind`, `saveAs`, `expect`, `origin`, `redirect`, and `outOfBand`). `{name}` is filled in a text or JSON `body`; a byte body (`Uint8Array`, `ArrayBuffer`, `Blob`) is an upload and is sent as it is, under the step's `mediaType`. Plus:
+
+| on a request step                                                                                   | meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `assert: { from, equals \| contains \| excludes \| omits \| atLeast, observes? }` or a list of them | The value at `from` — a JSON path (`[*]` fans out, `[?(@.id=='{id}')]` keeps the items whose field equals a value), or `header:Name` / `cookie:name` as `saveAs` reads them — must equal, contain, or (`excludes`) no longer hold the text, or (`omits`) appear in no node's text, an object's JSON text included (`'"token":'` is a key named `token` at any depth), or (`atLeast`) be a number no lower than it: a count from a sampled or eventually consistent store promises a floor, not an exact value. `{name}` is filled in both the path and the value. A list must all hold on the same exchange, so a response read only a few times (a download with a cap) is checked once. A pass marks the step's own operation and every operation in `observes` `verified`, with this exchange as the observation; `excludes` records it as an absence, `atLeast` the number it saw |
+| `poll: { within: "30s", everyMs?: 500 }`                                                            | Re-send until the assert passes or the window runs out — for reads that are eventually consistent. The wait starts at `everyMs` and doubles up to 5s, never below `everyMs`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| `anonymous: true`                                                                                   | Send without the scenario principal's credential                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+
+| step                                                                                 | meaning                                                                                                                               |
+| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
+| `{ external: "pay-with-test-card", input: { url: "{checkoutUrl}" }, as: "session" }` | Something only a browser or a person can do. The API tester calls `hooks.external(name, input)` and binds the returned string to `as` |
+
+A `saveAs` that lands on an object, a number or a boolean, or an `external` step that returns JSON, keeps the value whole: a body or `input` field that is exactly `"{name}"` sends that value with its type, not its string form. Inside a longer string it is interpolated as text. That is how a WebAuthn ceremony passes the server's options to a software authenticator and its credential back.
+
+`principal` picks the config principal the steps act as (default: the first), and a step's `principal` sends that one step as another; each must be a principal the config declares, or the config is refused before anything runs. A scenario that sends as a principal BLOCKED at sign-in, itself or in a step, is not run: it is `BLOCKED` by that principal, what it covers stays ungraded, and the run goes on to its report. Each step sends the credentials its operation's security requires; its `scope` parameters are bound from the first step. A record a step creates with a configured principal's credential is ledgered as that principal's and removed at teardown like any other. A step that sends a credential no configured principal holds — an account the scenario signed up itself — creates records the API tester does not track: cleaning those up is the scenario's job. A write's own response never proves that write — its observation has to come from a later read. Each id in `covers` must be called by a step — a scenario that covers what it never calls is refused before the run starts. A `{name}` a step reads must be filled by then: by the scenario principal's `scope`, a login `claims` key written `scope.<name>` (any scheme's, or the principal's own `security` override), or an earlier binding — a `bind` on the same step or before, an earlier step's `saveAs` or `saveClaimsFrom`, or an out-of-band or external `as`. An assert reads before its own step's `saveAs` binds. A scenario that reads a name nothing fills is refused when the config loads, naming the step and the name (`scenarios.reauth.steps[10] reads {orgld}`). A step that fails stops the scenario: the operation it blamed is `failed` with the exchanges as evidence, even when `covers` does not name it (a support step that arms an account fails as loudly as a covered one), and every covered operation it never reached is `blocked` with the step named. A step body field may hold a file as `{ bytes, filename, mediaType }`; on a multipart operation that field is sent as that part under that filename, so a scenario can upload one document under two names. An operation a scenario covers is never graded by the contract floor. A scenario with `origin` belongs to that configured origin: its steps and `covers` name that origin's operations, it runs in that origin's pass with the same principals and their scope, and its grades land on that origin's rows (`--ops files:file.put` targets it). A single step can also set `origin` to reach another origin as support; such a step never covers a main operation.
+
+## Environment interpolation
+
+After the module loads, every string in the config is scanned for `${NAME}`:
+
+```ts
+security: {
+	bearer: { static: "${API_TOKEN}" },
+}
+```
+
+If `API_TOKEN` is unset, the API tester exits with an error. Do not commit secrets; put them in the environment.
+
+Template literals in a `.ts` config (`Bearer ${process.env.API_TOKEN}`) are evaluated by Node _before_ the API tester sees the object. Either style works; `${NAME}` is what a `.json` config can use.
+
+`testing api doctor` and `testing api plan` send nothing, so they read a config whose variables are not set yet: an unset `env("NAME")` or `${NAME}` stays the literal `${NAME}`, and the command lists it. `testing api run` never does this, whatever `TESTING_API_DEFER_ENV` the shell exports, and refuses to send a credential that is still such a literal.
+
+Names match `[A-Z0-9_]+` case-insensitively.
+
+## Loading TypeScript configs
+
+`.js` / `.mjs` / `.json` load everywhere.
+
+`.ts` / `.mts` configs, such as the one `testing api init` writes, load under plain `node` on the API tester's floor, Node 22.18 or later, where type stripping is on by default. The published `the API tester` binary is itself JS; it still has to `import()` your config. If that fails (an older Node, or `--no-experimental-strip-types`), the error says so. Workaround: write `.mjs`.

package/docs/api/limits.md

@@ -0,0 +1,23 @@
+# Limits
+
+What the API tester cannot do yet, what you see when you hit it, and what to do instead. Each one is a gap the API tester reports, never a finding it invents.
+
+## Deliberate non-features
+
+These are by design. Do not look for a flag that turns them off.
+
+- **Default request timeout is 180s.** `network.requestTimeoutMs` (per attempt) aborts a stuck `fetch`, including a hung stream; `0` disables it. Watch `status=in_flight` and `idle_ms` in the progress log as that budget is consumed.
+- **A long `Retry-After` is not waited for.** A 429 is retried up to 5 times and a 503 that names its wait up to 3, each waiting out `Retry-After` up to `network.maxRetryAfterMs` (default 60s). A longer wait is neither waited for nor retried: the answer stands, and its finding names the wait and the limit. One 401 forces a credential refresh and a single retry; a second 401 is evidence.
+- **Network throws are not HTTP.** Offline, DNS, reset or timeout: 4 retries, then one wait (default 60s) for the link. If it does not come back, `net.unreachable` is recorded, the remaining work stands down, the report is still written, and the run exits `3`. Failed attempts are journaled as `status: 0` with `{ error: "network", kind }`.
+- **No `servers[]` at run time.** `testing api init` reads the first absolute server URL into `baseUrl`; a run always uses `baseUrl`. Extra hosts are `origins[]`, each with its own `spec`. An app origin that answers HTML and `Set-Cookie` is a request step with an absolute URL, not a dummy spec in `origins[]`.
+- **No OCR.** Multipart file parts are sent as dummy, pool, `each` or `resolveUpload` bytes. The API tester checks HTTP statuses and JSON answers, not whether a PDF is a real invoice.
+- **No webhook, callback or link-object following**, except the explicit auth or invite GET of a URL `resolveOutOfBand` returned. A hook-side `fetch` of that URL is not a recorded exchange.
+- **URL `$ref`s are not fetched.** In-document `$ref`s are, and so are relative refs to files inside the document's own folder.
+- **`x-idempotent` is not the idempotency check.** That check keys off a documented `Idempotency-Key` header.
+- **Rate-limit pacing covers only the API tester's own requests.** It cannot see other traffic to the backend, so a shared budget can still trip while the API tester's own share stayed within the declared rate.
+- **An equality-only filter grammar** cannot express `neq`, `gt`, `like`, `and` or `or`; those checks do not apply, they do not fail. `or()` is PostgREST-only.
+- **Leftover rows on a shared database** poison numeric and filter checks (`1, 10, 2` from old text-sorted rows beside a fresh numeric cohort). Wipe them between runs if a `--keep-fixtures` run or a crashed teardown left any.
+- **`--only` takes plan names** (`store`, not `stores` or `/v1/stores`).
+- **`--ops` trusts support operations.** A targeted green means the targets passed, given that what they needed worked. A defect in an untargeted operation is out of scope, even when a target called it.
+- **The default cohort is 7.** `pagination.limit-respects-documented-max` needs `cohortSize` above the documented maximum.
+- **One principal writes:** the one `writer` names, or the first. Further principals are peers and a rank lattice, not a pool of writers.

package/docs/api/migrating-0.25.md

@@ -0,0 +1,40 @@
+# testing api — Migrating to 0.25
+
+[← back to the overview](../../README.md)
+
+0.25 changes how a config says who the API tester acts as, and where its relative paths point. A config that still uses a removed field exits 2 and names the replacement, so nothing changes silently.
+
+## Relative paths resolve from the config's folder
+
+A config's relative `spec`, each origin's relative `spec`, and a relative `outDir` now resolve from the config's own folder, no longer from the working directory. A config that sat beside its document keeps working from any directory; one written for a working directory elsewhere needs its paths made relative to its own folder (or absolute).
+
+## Removed config fields
+
+A config with any of these exits 2, naming the replacement:
+
+| removed                                                      | instead                                                            |
+| ------------------------------------------------------------ | ------------------------------------------------------------------ |
+| `principals` as an array                                     | a map of id → identity data                                        |
+| `principals[].headers`                                       | `security.<scheme>: { static: … }`                                 |
+| `principals[].auth`                                          | `security.<scheme>: { login: { steps, from } }`                    |
+| `principals[].roots`, top-level `roots`                      | `principals.<id>.scope`, and `tenant`                              |
+| `rootsFromFlow`                                              | `security.<scheme>.login.claims`                                   |
+| `x-auth-flows`, in the config or the document                | a `security.<scheme>` recipe                                       |
+| a credential in `globalHeaders` (`Authorization`, `Cookie`…) | the scheme places it; `redact.headers` for a non-credential secret |
+| `hooks.resolvePrincipalAuth`                                 | `{ hook: true }` and `hooks.resolveCredential`                     |
+
+## The writer is named, or the first principal
+
+An array kept its order; a map's key order is easy to lose, and a sort-keys lint rule reorders it. The writer, the principal that makes the records every check works on, is the one top-level `writer` names, or else the first key of `principals`. A config whose principals a linter sorts, or that wants a writer other than its first entry, sets `writer: "<id>"`. `testing api doctor` prints who writes and why.
+
+## Renamed checks
+
+Three check ids are spelled en-US now:
+
+| 0.24                                   | 0.25                                  |
+| -------------------------------------- | ------------------------------------- |
+| `select.projection-honoured`           | `select.projection-honored`           |
+| `select.nested-honoured`               | `select.nested-honored`               |
+| `spec.declared-rate-limit-is-honoured` | `spec.declared-rate-limit-is-honored` |
+
+A baseline that names an old id exits 2 and names the new one, as it does for any id no check raises, so a known finding never reads as fixed because its check was renamed. `testing api diff` between a 0.24 report and a 0.25 one shows those checks' findings as fixed under the old id and new under the new one.