@workerstack/testing 0.0.0-stage → 0.31.0

package/docs/api/commands.md

@@ -0,0 +1,309 @@
+# testing api — Commands
+
+[← back to the overview](../../README.md)
+
+## Commands
+
+```
+testing api run          --config <file>     test a live backend and write a report
+testing api doctor       --spec <url|file>   what the API tester can and cannot test, and why
+testing api init         --spec <url|file>   write a starter testing.api.config.ts from the document's security schemes
+testing api plan         --spec <url|file>   print the derived model (offline)
+testing api serve        [--defects A,B]     run the demo API
+testing api conformance                      self-test: injected defects vs detection
+testing api baseline     --from <run>        write a known-findings baseline from a run's report
+testing api diff         <runA> <runB>       new, fixed and persisting findings between two runs
+testing api help
+```
+
+`--spec` for `doctor` / `plan` can be replaced by `--config` (the spec is read from the config). `--json` makes those two commands emit machine-readable output. `--base-url` on `doctor` / `plan` is only used to resolve a relative spec path.
+
+`--untagged` is a **serve** flag (and a conformance concern). It is not a `run` flag.
+
+`testing api help`, `testing api --help` and `--help` after any command print every command and flag, from the same table the command line is read with. An unknown flag or command, or a flag without its value, is refused with exit `2` and the help.
+
+### `testing api run`
+
+Requires `--config`. CLI flags override the same field in the config when both are set.
+
+| flag                                       | default                                  | meaning                                                                                                                                         |
+| ------------------------------------------ | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--config`                                 | required                                 | module or JSON file, default export                                                                                                             |
+| `--base-url`                               | `config.baseUrl`                         | backend origin                                                                                                                                  |
+| `--ops`                                    | `config.ops` or every operation          | comma-separated operationIds to grade; everything else is support. `*` globs, `<originId>:` prefixes. See [Targeted runs](#targeted-runs---ops) |
+| `--only`                                   | `config.only` or all entities            | comma-separated entity names as `testing api plan` prints them (singularized); each grades every operation it owns. Joins `--ops` as a union    |
+| `--seed`                                   | `config.seed` or `1`                     | fixture generation seed (reproducible)                                                                                                          |
+| `--out`                                    | `config.outDir` or `./.testing/api/runs` | history root; each run writes `<out>/<datetime>/` and updates `latest`                                                                          |
+| `--report`                                 | none                                     | comma-separated extra reports written into the run directory; `junit` writes `junit.xml`                                                        |
+| `--max-in-flight`                          | `config.maxInFlight` or `4`              | HTTP requests allowed at once                                                                                                                   |
+| `--keep-fixtures`                          | `config.keepFixtures` or false           | do not DELETE what the run created                                                                                                              |
+| `--keep-credentials`                       | false                                    | write the credentials the run held to `.testing/api/credentials.json` (mode 0600), outside `--out`; `principals.json` never holds one           |
+| `--quiet`                                  | false                                    | no stderr progress; files under `--out` still update                                                                                            |
+| `--save-exchanges` / `--no-save-exchanges` | on unless `--profile cheap`              | persist every HTTP exchange under the run dir (`exchanges.jsonl`, `exchanges/`, `blobs/`). Does **not** change in-memory transcript RAM         |
+
+**Exit codes:** `0` no defects, `1` at least one root-cause finding (`BACKEND_BUG`, `SPEC_BUG`, `SECURITY`, `AMBIGUITY`) **or**, on a targeted run, a target that `failed`, `2` usage error (missing `--config`, no principals, an `overlay` or `allow` entry that names nothing, unknown flag, a flag without its value or with a value of the wrong kind, an `--ops` / `--only` name that matches nothing, a target the profile excludes), `3` the API tester could not do what it was asked — the network never came back, `hooks.beforeRequest` refused a request (unless `fence.failRun` is false), the run graded no operation at all, a target was never judged, or the API tester itself failed, `4` the run may have left records behind — a record teardown could not show gone (its delete was refused or failed, it is still listed after a 2xx delete, its creator never saw it, or `hooks.verifyRemoved` found it), or a create answered 2xx with no id the API tester could read or confirm, so teardown could not remove it. Each is listed under **Leftovers**, and `4` takes precedence over every other outcome. An entity with no delete operation is reported as a gap and does not exit `4` on its own. A first Ctrl-C or SIGTERM stops the run in whatever phase it is in, removes what it created, writes the report with `interrupted: true`, and exits `130` (SIGINT) or `143` (SIGTERM); a second one exits `130` at once, without teardown. `COVERAGE_GAP` and `BLOCKED` do not fail a full run. `latest` moves to a run only once its reports are written.
+
+Example:
+
+```bash
+testing api run --config testing.api.config.ts --only store,product --out .testing/api/runs/prod
+```
+
+`--only store,product` matches the **entity names** from `testing api plan`, not path segments. `/v1/stores` is usually the entity `store`. An unknown name exits `2` with the nearest names.
+
+A run with no principals exits `2`. Isolation checks then need a second principal; they are skipped, not failed, when only one is present.
+
+### Targeted runs (`--ops`)
+
+A full run grades every operation. After changing two endpoints you usually want those two, not the whole API:
+
+```bash
+testing api run --config testing.api.config.ts --ops table.create,row.list
+testing api run --config testing.api.config.ts --ops 'row.*'            # glob within an operationId
+testing api run --config testing.api.config.ts --ops cdn:asset.get      # an operation on the origin with id "cdn"
+testing api plan --config testing.api.config.ts --ops table.create      # dry run: what would be graded, queued, called
+```
+
+**Targets** are graded: every check whose subjects include a target runs the full matrix against it. **Support** operations are called and never graded: principal sign-up, parent creates, the cohort create behind a list, reads used as an oracle, teardown deletes. A list route cannot be tested without rows in it, so its create still runs, but a defect in that create is out of this run's scope.
+
+How the scope is decided:
+
+- Every check declares its **subjects**, the operations whose contract it judges (`filter.*` judges the list route, `patch.minimality` the update, `effects.declared-effect-occurs` each `x-effects` operation). A check runs when one of its subjects is a target. Others are **out of scope**: not run, not skipped.
+- An entity is seeded only when one of its checks grades a target. Declared edges pull in the other direction too: targeting a read route that another entity's create names in `x-invalidate` queues that entity, so its invalidation check grades the route.
+- Checks that judge several operations (`effects.*`, `async.*`, `response.status-is-documented`, `schema.error-response-matches-document`, `invalidation.declared-route-changes`) invoke and judge only the targeted ones. An untargeted `x-effects` upload is never POSTed.
+- Every finding names the operations it judges (`operations` in `report.json`).
+- An operation counts as graded only when a check exercised it. A check that judges after the fact (`response.status-is-documented` reads the transcript) grades only the operations it saw. A check that reports a coverage gap for an operation did not grade it. `--only <entity>` therefore surfaces every action no check can reach as `untested`.
+
+Each target ends with one status:
+
+| status         | meaning                                                                                                                                                 |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `passed`       | at least one check graded it and none failed                                                                                                            |
+| `failed`       | a defect finding is attributed to it                                                                                                                    |
+| `inconclusive` | checks ran and none could decide                                                                                                                        |
+| `blocked`      | every check was suppressed, or a support operation it needed failed (named, with evidence)                                                              |
+| `untested`     | nothing exercised it. The reason says why (`unmodeled`, `entity-not-testable`, `coverage gap: …`, `no check applied: needs …`, `no check exercised it`) |
+
+A targeted run exits `0` only when every target is `passed`. A target nothing could grade must not read as a pass.
+
+Validation happens before any principal signs up: a name that matches nothing exits `2` with the nearest operationIds; an exact target the active `--profile` excludes exits `2`; a glob or `--only` match the profile excludes is dropped and listed. An origin no target names is not run.
+
+A targeted run is for iterating. It grades what you named and what those operations observably affect through declared edges. It does not catch a regression in an untargeted operation caused by shared code, so run the full suite before a release.
+
+### `testing api doctor`
+
+Offline. Loads the document, builds the model, prints coverage.
+
+```bash
+testing api doctor --spec https://api.example.com/openapi.json
+testing api doctor --config testing.api.config.ts --json
+testing api doctor --spec ./openapi.yaml --base-url https://api.example.com
+```
+
+Human output:
+
+- `trackable` — entities with an identity and a read surface
+- `listable` — those that also have a list (query checks need this)
+- tags that are absent, and the checks each tag would unlock
+- tags that would **sharpen** checks that already run (`x-query`, `x-tenant`)
+- per-operation gaps (`x-entity` could not be inferred, assumed tenant param, …)
+- what the document declares but the API tester never calls, one gap each and never blocking: `webhooks` (one per webhook, named `webhook <name>`), an operation's `callbacks`, a response's `links` (naming the status), and OpenAPI 3.2 `additionalOperations`
+- external `$ref`s that were not fetched
+- `security` — each scheme the document declares: its kind, where its credential goes (`header Authorization: Bearer {credential}`), and, with `--config`, the recipe that makes it (`login`, `static · format`, `static · +1 placement`, or `no recipe`). A scheme the API tester cannot send says why instead.
+- `requirements` — the operations grouped by what they need, most first: `|` is any one alternative, `+` every scheme of one together, `[read,write]` the scopes, and `anonymous` sends nothing. With `--config`, each group names the principal that sends it: the first in config order whose recipes satisfy it, decided as a run decides it (`alpha`, `alpha via jwt` when there is a choice, `beta (alpha: <why not>)`, or `none — <why>`).
+- `no recipe` (with `--config`), `unsupported` and `undeclared` — schemes a run cannot send, each with the operations that need only it.
+- roots — path parameters no create operation makes. A **prefix root** (`/projects/{project_id}/…`: tagged `x-root`, or followed by another parameter or by its entity's collection) is the config's to supply: in every principal's `scope`, a login's `scope.<parameter>` claim, or `hooks.resolvePathValue`. Doctor lists those the config supplies and how, and those it still must. An **unmade id** (`/invites/{token}/accept`, `/grants/{grant_id}`) is one a flow yields during a run; it is listed and never blocks. In JSON: `prefixRoots: { required, supplied }` and `unmadeIds`.
+- `env` — with `--config`, the variables the config reads through `env()` or `${NAME}` that are not set. Nothing is sent, so `doctor` reads such a config and lists them; `run` stops at the first.
+
+`--json` shape:
+
+```json
+{
+	"blocking": 1,
+	"entities": 12,
+	"trackableEntities": 10,
+	"testableEntities": 10,
+	"listableEntities": 8,
+	"prefixRoots": {
+		"required": ["organization_id"],
+		"supplied": [{ "root": "project_id", "by": "principal scope" }]
+	},
+	"unmadeIds": ["token"],
+	"externalRefs": ["https://example.com/shared.yaml"],
+	"gaps": [{ "operationId": "table.list", "tag": "x-query", "detail": "…" }],
+	"security": {
+		"schemes": [
+			{
+				"name": "jwt",
+				"kind": "bearer",
+				"bearerFormat": "JWT",
+				"deprecated": false,
+				"placement": { "in": "header", "name": "Authorization", "template": "Bearer {credential}" },
+				"extraPlacements": 0,
+				"recipe": "login"
+			}
+		],
+		"operations": [
+			{
+				"operationId": "row.list",
+				"source": "operation",
+				"alternatives": [[{ "scheme": "jwt", "scopes": [] }]],
+				"satisfiedBy": { "principal": "alpha", "alternative": 0 }
+			}
+		],
+		"noRecipe": [{ "name": "legacy", "operations": 2 }],
+		"unsupported": [
+			{ "name": "mtls", "operations": 1, "why": "mutualTLS is a TLS client certificate, not a request credential" }
+		],
+		"undeclared": [{ "name": "ghost", "operations": 3 }]
+	},
+	"unsetEnv": ["TESTING_API_JWT_TOKEN"]
+}
+```
+
+`satisfiedBy`, `alphaUnsatisfied` (the first principal's reason when a later one satisfies the operation), `unsatisfied`, `noRecipe` and `unsetEnv` are there only with `--config`.
+
+Exit `1` if there are **blocking** gaps: entities that are not trackable (no identity / no read), or a prefix root nothing in the config supplies. Advisory gaps (missing `x-query`, no `x-async`) print and still exit `0`. With `--config`, an operation no principal can send also blocks. Unset variables never do: they are a note.
+
+### `testing api init`
+
+Offline. Reads the document and writes a starter config from its security schemes.
+
+```bash
+testing api init --spec ./openapi.json
+testing api init --spec https://api.example.com/openapi.json --out testing.api.config.mjs
+```
+
+| flag      | default                   | meaning                                                                                   |
+| --------- | ------------------------- | ----------------------------------------------------------------------------------------- |
+| `--spec`  | required                  | the document, a URL or a path; written into the config exactly as given                   |
+| `--out`   | `./testing.api.config.ts` | the config written: `.ts`, `.mts`, `.js` or `.mjs` (a `.json` config cannot call `env()`) |
+| `--force` | false                     | overwrite an existing `--out` file                                                        |
+
+The config has one `security` recipe per scheme the document declares, each reading its secret with `env()`, and one principal, `alpha`. Bearer, OpenID Connect and API-key schemes get a static token or key, basic gets a username and password pair, and OAuth 2 gets the client-credentials or password grant the API tester performs itself, else a static token. A comment above each recipe says where the credential goes, the scopes operations ask of it (`holds`), and a sign-in operation the document offers. `baseUrl` is the first absolute `servers` URL, else `env("TESTING_API_BASE_URL")`. A scheme the API tester cannot send (mutualTLS, http digest, an unknown type) gets no recipe, only a comment naming its type and why. A document with no scheme gets a commented `overlay.securitySchemes` stub instead, since it may be truly anonymous.
+
+It prints the file written, every env var the config reads (`set TESTING_API_BASE_URL, TESTING_API_JWT_TOKEN`), each scheme it left out, and the next step, `testing api doctor --config <out>`. Exit `2` without `--spec`, for any other `--out` extension, on an existing file without `--force` (the file is untouched), and on a document the API tester does not read (Swagger 2.0, OpenAPI 3.3+).
+
+### `testing api baseline`
+
+```
+testing api baseline --from .testing/api/runs/latest [--out testing.api.baseline.json] [--force]
+```
+
+Reads `<run>/report.json` (`latest` is followed) and writes one entry per root-cause finding — `check`, `entity`, the first operation it judges, and its summary as `reason` — deduplicated and sorted. Edit the reasons to point at tickets. An existing `--out` file is kept unless `--force`. Exit `2` when the run cannot be read.
+
+Point `baseline` in the config at the file. A finding an entry matches is reported under **Known** and does not fail the run; every other root cause still does; an entry that matched nothing is reported under **Fixed**, never as a failure. An entry with `operationId` matches only findings that judge that operation; without it, the check's findings on that entity (or every entity, without `entity`) — so keep entries narrow, or a new defect of the same check on another operation is hidden.
+
+### `testing api diff`
+
+```
+testing api diff .testing/api/runs/20261015T210304Z .testing/api/runs/latest [--json]
+```
+
+Compares two runs' `report.json`: root-cause findings that are **new**, **fixed** and **persisting**, matched by a key that ignores run-specific values (record ids, UUIDs, timestamps, ports, nonces) and the wording of the summary; operations whose status changed; and both runs' target versions. A report without `scope` makes coverage "not comparable". Always exits `0` — it is a report, not a gate — except `2` when a run cannot be read.
+
+### `testing api plan`
+
+Offline. Prints the derived entity graph, operations, and query capability.
+
+```bash
+testing api plan --spec ./openapi.yaml
+testing api plan --config testing.api.config.ts --json
+```
+
+Human columns:
+
+```
+entity              CLRUD  ident      read surface
+store               CLRU·  id         2 route(s) (inferred)
+                                      GET /v1/stores
+                                      GET /v1/stores/{store_id}
+```
+
+`CLRUD` is Create / List / Read / Update / Delete. `·` means that slot is missing. `ident` is the identity property. Read surface is declared (`x-invalidate`) or inferred (sibling collection/item routes).
+
+`--json` is `{ entities, operations, prefixRoots, unmadeIds, security, unique }` (roots split as `doctor` splits them) — the full `SpecModel` maps, including conventions, query capability, async, invite, and path params. Use this when you need to know what the API tester will call something.
+
+A `security — operation · requirement · principal` section follows the table: every operation, by id, with its requirement and, with `--config`, the principal that sends it. With `--ops` / `--only`, each target row ends with `<requirement> (<principal>)`. `--json` adds the same `security` object as `doctor`, and with `--config` an `unsetEnv` list; the unset variables also print as one `env` line on stderr, so stdout stays the plan.
+
+With `--ops` / `--only` (flag or config), `plan` prints the targeted scope instead: each target with its entity and the checks that would grade it, untestable targets with the reason, the entities that would be queued, and the support operations predicted from the graph. `--json` adds a `scope` object with the same fields. An unknown name exits `2`.
+
+### `testing api serve`
+
+In-process demo API. Same fixture as conformance.
+
+```bash
+testing api serve
+testing api serve --defects STALE_LIST,PATCH_REPLACES
+testing api serve --backend sqlite --dialect classic
+testing api serve --untagged
+```
+
+| flag         | default     | meaning                                                                                                            |
+| ------------ | ----------- | ------------------------------------------------------------------------------------------------------------------ |
+| `--backend`  | `memory`    | `memory` \| `sqlite` \| `postgres`                                                                                 |
+| `--dialect`  | `postgrest` | `postgrest` \| `classic` \| `linked` \| `jsonapi` \| `plain`                                                       |
+| `--defects`  | none        | comma-separated names from [Reference defects](reference-defects.md#reference-defects-testing-api-serve---defects) |
+| `--untagged` | false       | serve the same API behind a spec with every `x-*` tag stripped                                                     |
+
+Printed keys: `key_alpha` (tenant `proj_alpha`), `key_beta` (tenant `proj_beta`). Spec: `{url}/v1/openapi/spec`. Stop with ctrl-c.
+
+`labs/api/local.config.ts` is written for this server.
+
+Dialects are **reference-backend shapes**, not something you configure against your API. They exist so conformance proves checks read the document rather than one fixture's spelling:
+
+| dialect     | filter                      | sort       | select           | page model          | envelope                          |
+| ----------- | --------------------------- | ---------- | ---------------- | ------------------- | --------------------------------- |
+| `postgrest` | `filter=status.eq.active`   | `name.asc` | `select=id,name` | `page` + `cursor`   | entity-named + `count`/`hasMore`  |
+| `classic`   | `filter=status=eq:active`   | `sort=`    | `fields=`        | `page` + `per_page` | `{ data, total_count, has_more }` |
+| `linked`    | postgrest                   | dotted     | `fields=`        | `offset` + `limit`  | raw array + `Link: rel=next`      |
+| `jsonapi`   | postgrest                   | `-name`    | `fields[table]=` | `page` + `size`     | `{ data, total, has_more }`       |
+| `plain`     | `?status=active` (equality) | `name:asc` | `fields=`        | `page` + `limit`    | `{ items, total, has_more }`      |
+
+`postgres` needs the `postgres` driver, an optional peer dependency the API tester does not install (`npm install postgres`, `^3.4`), and a server on the default `postgres` database (local, default driver connection). Without the driver, `serve --backend postgres` exits `2` saying so. `sqlite` needs Node's `node:sqlite`, built in with no flag on the API tester's floor, Node 22.18. Missing backends fail at serve time rather than falling back.
+
+### `testing api conformance`
+
+Self-test. Not for your API. Injects named defects into the reference backend and asserts the API tester reports the matching check.
+
+```bash
+# this repo
+npm test
+
+# after install; Node 22.18 or later runs every local backend, sqlite included, with no flag
+testing api conformance
+testing api conformance --backend memory --dialect plain
+testing api conformance --fuzz 300 --max-defects 12 --seed 7
+testing api conformance --precision 60 --backend memory
+testing api conformance --parser
+testing api conformance --backend d1
+testing api conformance --only STALE_LIST,PATCH_REPLACES
+```
+
+| flag              | meaning                                                                                                                            |
+| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
+| `--backend`       | `memory` \| `sqlite` \| `postgres` \| `d1`. Default: every _local_ backend that is available. `d1` is never default — it is remote |
+| `--dialect`       | pin one shape; default runs postgrest on each backend plus classic/linked/jsonapi/plain on memory                                  |
+| `--fuzz [n]`      | random _sets_ of defects (default 25 if flag is bare)                                                                              |
+| `--max-defects`   | cap per fuzz combination (default 4)                                                                                               |
+| `--precision [n]` | vary _data_ against a correct backend; any finding is a false positive (default 50 if flag is bare)                                |
+| `--seed`          | replay a fuzz/precision run                                                                                                        |
+| `--parser`        | only the hostile-document + example-spec + tag-unlock suites                                                                       |
+| `--only`          | restrict injected defects (comma-separated `STALE_LIST,…`)                                                                         |
+| `--jobs`          | worker processes the legs share (default: CPUs − 1, at most 4)                                                                     |
+| `--legs`          | run only these parts: `suites`, `passes`, `shapes`, `recall`, `scope`, `fuzz` (comma-separated)                                    |
+| `--shapes`        | the shapes the `shapes` and `recall` legs judge behind (comma-separated); all when absent                                          |
+| `--proven-out`    | write the checks this run's defects proved to a JSON file, for `--aggregate`                                                       |
+| `--aggregate`     | judge a folder of `--proven-out` files together; exit `1` on any check no file proved                                              |
+| `--fuzz-seed`     | seed of the 40-combination pass a full run ends with (default 1); CI passes a fresh one                                            |
+| `--skip-backend`  | backends a full run may leave out when unreachable (comma-separated); otherwise a missing one fails the run                        |
+
+A default `testing api conformance` runs every leg: the suites (hostile documents, the `labs/api/annotated-openapi.yaml` model lock, the tag-unlock map and the other self-contained suites), the passes on every local engine plus the extra dialects on memory, the correct-backend shapes, every defect again behind each shape that runs clean (recall), `--ops` recall from the default pass's own runs, a 40-case combination pass, and finally the checks no defect proved. The extra dialects run only the defects whose symptom depends on how a listing is asked for and answered.
+
+Selecting a `--backend`, `--dialect` or `--only` runs the suites and those passes alone; `--legs` runs exactly the legs named (the `scope` leg brings the default memory pass it judges from); `--parser` runs the suites alone. Each pass prints its time, recall prints a line per shape as it finishes, and the run ends with the largest worker's memory. Set `TESTING_API_HEAP_SNAPSHOTS=<dir>` to write a worker heap snapshot after every task.
+
+D1 needs `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_D1_DATABASE_ID`, `CLOUDFLARE_API_TOKEN`; it never runs unless asked for by name. Postgres needs the optional `postgres` driver (`npm install postgres`, `^3.4`) and a reachable server on the default connection (`database: "postgres"`). A full run fails on a local backend it cannot reach unless `--skip-backend` names it; for Postgres it says whether the driver is not installed or no server answered. `--backend postgres` on an install without the driver exits `2`. Exit `1` if any suite, pass, shape, recall, fuzz or precision case fails, or a full run leaves a check unproven.
+
+#### Sharded
+
+`bun run conformance:sharded [--jobs n] [--skip-backend sqlite,postgres] [--out dir]` runs the same self-test as short legs, one process each — the suites, each pass, recall a few shapes at a time, `--ops` scope, the combination pass — writing each leg's proven checks with `--proven-out`, then judges them with `--aggregate`. It prints every leg's exit and time. No leg holds the whole matrix, so a machine short of memory still finishes. `--list` prints the legs that need no server as JSON, which CI runs as a matrix, and `--leg <name> --out dir` runs one leg alone, as a CI job or to rerun a failure; flags after `--` go to that leg. The legs are `src/api/conformance/shards.ts`.

package/docs/api/configuration/hooks.md

@@ -0,0 +1,162 @@
+# testing api — Hooks
+
+[← back to the overview](../../../README.md)
+
+## Hooks
+
+```ts
+hooks: {
+  // Return null to retry (attempt is 1-based). The API tester backs off until a value arrives.
+  // `scope` is the flow at this step. No credential is passed: one travels only through the API tester's client.
+  // `since` (unix ms) is when the request that named `address` was sent, when one did:
+  // a mail delivered before it answers some earlier request, so return null and wait.
+  resolveOutOfBand: async ({ address, kind, attempt, scope, since }) => {
+    const mail = await readMailCatcher(address, kind)
+    if (mail === null || (since !== undefined && mail.receivedAt < since)) return null
+    return mail.link // URL or token string
+  },
+  // Remove a principal this run provisioned. `sends` places its last live credential as its scheme does;
+  // `stale` says why it could not be renewed, when it could not.
+  teardownPrincipal: async (address, { credential, sends }) => {
+    await fetch(`https://api.example.com/v1/auth/account`, { method: "DELETE", headers: sends.headers })
+  },
+  // A path parameter the API tester cannot create. Told the principal it acts as, its scope and identity, and the
+  // credential and placement the operation would carry as it: decode the token, or call the API as it.
+  resolvePathValue: async ({ param, principal, credential, sends }) => {
+    if (param !== "project_id" || credential === undefined) return null
+    const res = await fetch("https://api.example.com/v1/projects", { headers: sends?.headers })
+    return (await res.json())[0]?.id ?? null
+  },
+  // Return a file to send, `{ fields }` to replace the whole request, or null to fall through.
+  resolveUpload: async ({ operationId, field, contentMediaType }) => {
+    if (operationId === "extract.once" && field === "file") {
+      const bytes = await Deno.readFile("./invoices/known.pdf")
+      return { bytes, filename: "known.pdf", mediaType: "application/pdf" }
+    }
+    return null
+  },
+  // After globalHeaders; never a credential. Return null to add nothing.
+  resolveHeaders: async ({ operationId, method, url }) => {
+    if (operationId === "auth.register" || operationId === "auth.login") {
+      return { "cf-turnstile-response": await harvestTurnstile() }
+    }
+    return null
+  },
+  // Replace a generated JSON field. Null keeps the generator.
+  resolveInput: async ({ operationId, field }) => {
+    if (operationId === "billing.subscribe" && field === "payment_method_id") {
+      return process.env.STRIPE_TEST_PM
+    }
+    return null
+  },
+  // The credential for a `{ hook: true }` recipe, per scheme and principal. Called on every
+  // acquire, so a TOTP code is fresh. Null blocks that principal for that scheme.
+  resolveCredential: async (scheme, { id, identity }) => {
+    const pair = await readHarvested(scheme, id)
+    return pair === null ? null : { value: pair.access_token, expiresIn: pair.expires_in }
+  },
+  // After seed: add or replace any axis of the query catalog. Null keeps the merge.
+  resolveQueryCapabilities: async ({ entity, get }) => {
+    if (entity !== "row") return null
+    const body = await get("table.get")
+    return { filterable: /* harvest from body */ [] }
+  },
+  // A scenario `external` step: something only a browser or a person can do. Return the value
+  // the next step needs, or null to fail the step.
+  external: async (name, input) => {
+    if (name === "pay-with-test-card") return await payInBrowser(String(input.url), "4242424242424242")
+    return null
+  },
+  // Optional extra stop condition while x-wait polls.
+  awaitSideEffect: async ({ operationId, record }) => {
+    if (operationId !== "webhook.deliver") return null
+    return Array.isArray((record as { items?: unknown }).items) && (record as { items: unknown[] }).items.length > 0
+      ? true
+      : null
+  },
+}
+```
+
+Without `resolveOutOfBand`, an `outOfBand` step cannot complete. The API tester polls the hook; the hook must not sleep. Returning `""` is treated like `null`. The hook is not a recorded hop: if the human GETs a mailed URL, a later RequestStep must GET that URL so it appears in `exchanges.jsonl` and can `expect` / `saveAs`.
+
+Default schedule (0.6.2, unchanged unless `outOfBand` is set): **6** attempts, first sleep **200** ms, doubling, cap **3000** ms. The API tester sleeps after every miss, including the last, so the worst-case wait is
+
+`200 + 400 + 800 + 1600 + 3000 + 3000 = 9000` ms.
+
+That is too short for real mail (often 10–60 s) and for a human finishing Google OAuth or a Turnstile harvest. Configure it:
+
+```ts
+outOfBand: { attempts: 20, initialMs: 1000, maxMs: 8000 }
+// worst case: 1000 + 2000 + 4000 + 8000×17 = 143000 ms
+```
+
+Worst-case wait is `sum_{i=0}^{attempts-1} min(initialMs × 2^i, maxMs)`. `worstCaseWaitMs()` from the package computes it. Existing configs that omit `outOfBand` do not slow down.
+
+Without `teardownPrincipal`, an account no delete removes is reported as a leftover rather than cascade-deleted. Per-record DELETE still runs for seeded rows when a delete (or `x-cleanup`) exists. The hook is called with the last live credential (`ctx.credential`, and `ctx.sends`: its headers, query and cookies as the scheme places them), not an already-cleared one, so authenticated delete-account is expressible without a tester-key god route. Teardown renews that credential first. When it cannot (a refused refresh, or a login that would sign the account up again), the hook still gets the one the run held, with `ctx.stale` set to why. That credential has likely lapsed, so a hook that can reach the account another way, by its password or an admin, should take that way. One-argument JavaScript callbacks still run.
+
+### Proving a record gone
+
+Teardown counts a record gone when its delete answers 2xx and its list no longer holds it. A record its creator could never read back proves nothing that way, so teardown reads it once before the delete; one still never seen is a leftover (`its creator could never read it back, so teardown cannot tell whether it is gone`) unless an ancestor was removed earlier in the run. `hooks.verifyRemoved` settles it from somewhere HTTP does not reach:
+
+```ts
+hooks: {
+  verifyRemoved: async ({ entity, id, scope, principal }) => {
+    const row = await db.oneOrNone(`select 1 from ${entity}s where id = $1`, [id])
+    return row === null ? "absent" : "present"
+  },
+}
+```
+
+It is asked about every record teardown counted removed and every one it could not show gone, four at a time. `"present"` makes the record a leftover (`hooks.verifyRemoved says it is still present`), `"absent"` clears one never seen, and `"unknown"` keeps teardown's own verdict. A hook that throws counts as `"unknown"`, and the run reports one `teardown.verify-hook` gap.
+
+`resolveHeaders` is called on every dispatch (including the 401 retry). Merge order: `globalHeaders` → hook → per-request headers. The credential is placed by its scheme after them; a header from any of them that names a credential's place is an error, never an override. Use `ctx.operationId` / `ctx.method` / `ctx.url` to attach a one-shot captcha only on captcha ops. The API tester does not speak Turnstile.
+
+`resolveInput` is the JSON twin of `resolveUpload`. Return a value to replace that field (`payment_method_id` on `billing.subscribe`); `null` keeps the generator. `request.value` is the generated value, or `undefined` when the field was left out. Return `null` for an omitted field or a value outside the schema's length or range bounds so a constraint check is the body the server receives.
+
+`resolveQueryCapabilities` runs once per entity after seed. `get(operationId)` (or `"GET /path"`) uses the seeded parent scope so a follow-up read can harvest dynamic columns. A provided `filterable` / `sortable` / `searchable` / `selectable` list **replaces** that axis; omitted axes stay. JSON configs have no hook.
+
+`resolveCredential` is documented under [Credentials from a hook](principals-and-security.md#credentials-from-a-hook); `awaitSideEffect` with `x-wait`.
+
+Worked `query` overlay for a generic PostgREST-shaped API (not a named product):
+
+```ts
+export default defineConfig({
+	spec: "https://api.example.com/openapi.json",
+	baseUrl: "https://api.example.com",
+	security: { bearer: { static: `${process.env.TOKEN}` } },
+	principals: { alpha: {} },
+	query: {
+		operators: ["eq", "neq", "gt", "gte", "lt", "lte", "in", "nin", "like", "ilike", "is"],
+		operatorsByType: {
+			string: ["eq", "neq", "like", "ilike", "in", "nin", "is"],
+			number: ["eq", "neq", "gt", "gte", "lt", "lte", "in", "nin", "is"],
+			date: ["eq", "neq", "gt", "gte", "lt", "lte", "is"],
+			boolean: ["eq", "neq", "is"],
+		},
+		aliases: { ne: "neq" },
+		emptyIn: "match-none",
+		maxInValues: 100,
+		maxFilterConditions: 20,
+		searchEmpty: "match-all",
+		sort: { nulls: ["first", "last"], maxKeys: 3 },
+		select: { nested: false, unknown: "reject" },
+	},
+})
+```
+
+New surface still has to be listed. The API tester will not silently enable `in` / `ilike` / `is` / `contains` / search modes / `nullsfirst` on every PostgREST-shaped document.
+
+## Request fence
+
+`hooks.beforeRequest(request)` sees every request before it leaves the process — the document fetch, sign-in, seeding, checks, scenarios, teardown and the reachability probe — and answers `{ allow: true }` or `{ allow: false, reason }`. `request` is `{ method, url, headers, body, operationId?, phase, principal?, schemes?, forged? }`, with `phase` one of `load`, `auth`, `seed`, `assertion`, `probe`, `teardown` or `request`. `principal` and `schemes` name whose credential the request carries and under which security schemes, `forged: true` marks a forged value a probe sends, and all three are absent for an anonymous request, so a fence can decide by who and which scheme without parsing `Authorization`. beforeRequest sees the request as sent, including placed credentials; never log it unredacted.
+
+A refused request is never sent and never judged: its check is unresolved with a `fence` blocked entry, a refused seed blocks the entity, a refused delete is a leftover whose reason starts with `fenced`, and a refused sign-in or document fetch stops the run with exit 3. The report's **Fenced requests** section and `fenced` in `report.json` list every refusal. Any refusal fails the run (exit 3) unless `fence: { failRun: false }`.
+
+```ts
+hooks: {
+	beforeRequest: ({ method, operationId }) =>
+		method === "DELETE" && operationId !== "row.delete"
+			? { allow: false, reason: "only rows may be deleted on the shared backend" }
+			: { allow: true },
+},
+```

package/docs/api/configuration/origins-and-uploads.md

@@ -0,0 +1,85 @@
+# testing api — Origins and uploads
+
+[← back to the overview](../../../README.md)
+
+## Secondary origins
+
+One run, one primary `baseUrl`. A CDN (or any second host) keeps its own OpenAPI.
+
+```ts
+export default defineConfig({
+	spec: "https://api.example.com/openapi.json",
+	baseUrl: "https://api.example.com",
+	security: {
+		jwt: { login: { steps: [{ operationId: "auth.login", body: { email: "{principal.email}" } }], from: "$.token" } },
+	},
+	principals: { alpha: { email: "alpha@example.test", tenant: "acme" } },
+	origins: [
+		{
+			id: "cdn",
+			baseUrl: "https://cdn.example.com",
+			spec: "https://cdn.example.com/openapi.json",
+			security: { bearer: "jwt" },
+		},
+	],
+})
+```
+
+Each origin's document is read once, at load, and the config is checked against it too. An origin scheme sends a recipe's credential only where `origins[].security` maps it (`{ <origin scheme>: <recipe> }`); an unmapped scheme sends nothing, and an origin scheme that shares a name with a primary recipe must be mapped. A recipe is usually a primary scheme's. It may instead name a scheme only an origin declares, mapped there: the API tester checks it against that origin scheme, acquires it once for the run, and sends it in that scheme's place with the recipe's own `format` and `placements`. When two origins map the same such recipe, the first in `origins` names its scheme. The same principals run the matrix against that document. Login steps may set `origin: "cdn"` to send one hop to a named origin.
+
+Do not merge CDN routes into the API gateway document.
+
+## Uploads
+
+Multipart and binary parts are filled in this order:
+
+1. `hooks.resolveUpload` — a non-null `UploadFile` wins. `{ fields }` that includes a file part replaces the whole request. `{ fields }` that omits the file part overlays scalars and keeps the each / pool / dummy bytes.
+2. `uploads.each` fixture for this invocation, when that operation is listed.
+3. `uploads.pool` — first file whose extension / sniffed type matches the part's `contentMediaType`. Same `seed` + field + index → same pick. Ops not in `each` stay pick-one.
+4. A tiny dummy with sniffable magic (`%PDF-1.1`, 1×1 PNG, empty zip, …). Unknown types become 16 octet-stream bytes, not a skip.
+
+`uploads.each` is a matrix, not a source. `operationId → globs` means that operation is invoked once per matched file (after `eachMax`). Same seed does **not** collapse `each`. A hook that ignores `request.fixture` and always returns the same file will send that file N times.
+
+Each invocation is still a citeable hop (method, URL, status, fixture name, content-addressed body). Live `FormData` / `Blob` payloads are **not** kept on `Client.transcript` for the rest of the run — they are replaced with `{ sha256, bytes, mediaType }` (and multipart `{ parts }`) after the hop is journaled, or immediately when the journal is off. Fan-out therefore multiplies requests, not retained fixture bytes.
+
+```ts
+export default defineConfig({
+	spec: "openapi.json",
+	baseUrl: "https://api.example.com",
+	principals: [/* … */],
+	uploads: {
+		pool: ["./fixtures/**/*"],
+		each: {
+			"extract.once": ["./fixtures/**/*"],
+			"extract.stream": ["./fixtures/**/*"],
+		},
+		eachMax: 24,
+	},
+	hooks: {
+		resolveUpload: async ({ operationId, field, fixture }) => {
+			if (operationId === "extract.once" && field === "file") {
+				return { fields: { columns: "vendor,date,amount" } }
+			}
+			return null
+		},
+	},
+})
+```
+
+| case                            | outcome                                                     |
+| ------------------------------- | ----------------------------------------------------------- |
+| `each` omitted                  | pick-one (today)                                            |
+| glob matches 0 files            | warn once, no extra invocations, fall through to pool/dummy |
+| one path in the list is missing | drop that slot, warn once, never `BACKEND_BUG`              |
+| `eachMax` < match count         | first `eachMax` after sort, warn that it capped             |
+| fixture unreadable              | drop that slot, warn once, not `BACKEND_BUG`                |
+
+A missing pool path warns once and falls through. An empty pool match uses a dummy. The run does not fail. JSON configs may set `pool`, `each`, and `eachMax`. `resolveUpload` is TypeScript.
+
+`--profile cheap` (or any profile that excludes the op) still drops the whole family. `each` does not punch through a profile.
+
+Findings from an `each` invocation carry `fixture: "invoice.pdf"` and render as `extract.once · invoice.pdf`. One 5xx is one finding on that file.
+
+The API tester does not OCR. It sends bytes and checks HTTP / JSON. A 200 with empty extract rows is not automatically a backend defect. A 4xx because the dummy is “not a real invoice” is not automatically a backend defect if the dummy matched the declared `contentMediaType`.
+
+Prefer `multipart/form-data` when the operation documents it, even if JSON is also listed. Text form fields still use the string generator (`format` / `pattern` / `maxLength`). A part that is either text or file is sent as a file when the schema is binary.