@lovrozagar/oat 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1270 -178
- package/dist/cli.js +11 -16
- package/dist/index.d.ts +2 -2
- package/dist/index.js +1 -1
- package/dist/reference/defects.d.ts.map +1 -1
- package/dist/reference/defects.js +4 -1
- package/dist/reference/defects.js.map +1 -1
- package/dist/reference/dialect.d.ts.map +1 -1
- package/dist/reference/dialect.js +4 -1
- package/dist/reference/dialect.js.map +1 -1
- package/dist/reference/http.d.ts.map +1 -1
- package/dist/reference/http.js +15 -22
- package/dist/reference/http.js.map +1 -1
- package/dist/reference/model.d.ts.map +1 -1
- package/dist/reference/model.js +27 -3
- package/dist/reference/model.js.map +1 -1
- package/dist/reference/query.d.ts.map +1 -1
- package/dist/reference/query.js +19 -35
- package/dist/reference/query.js.map +1 -1
- package/dist/reference/spec.d.ts.map +1 -1
- package/dist/reference/spec.js +8 -6
- package/dist/reference/spec.js.map +1 -1
- package/dist/reference/store-api.d.ts.map +1 -1
- package/dist/reference/store-api.js +4 -1
- package/dist/reference/store-api.js.map +1 -1
- package/dist/reference/stores/memory.d.ts.map +1 -1
- package/dist/reference/stores/memory.js +1 -3
- package/dist/reference/stores/memory.js.map +1 -1
- package/dist/reference/stores/postgres.d.ts.map +1 -1
- package/dist/reference/stores/postgres.js +8 -8
- package/dist/reference/stores/postgres.js.map +1 -1
- package/dist/reference/stores/sqlite-driver.d.ts.map +1 -1
- package/dist/reference/stores/sqlite-driver.js +2 -4
- package/dist/reference/stores/sqlite-driver.js.map +1 -1
- package/dist/reference/stores/sqlite.d.ts.map +1 -1
- package/dist/reference/stores/sqlite.js +22 -27
- package/dist/reference/stores/sqlite.js.map +1 -1
- package/dist/report/console.d.ts.map +1 -1
- package/dist/report/console.js +4 -4
- package/dist/report/console.js.map +1 -1
- package/dist/report/matrix.d.ts.map +1 -1
- package/dist/report/matrix.js +57 -28
- package/dist/report/matrix.js.map +1 -1
- package/dist/report/render.d.ts.map +1 -1
- package/dist/report/render.js +12 -20
- package/dist/report/render.js.map +1 -1
- package/dist/runtime/async.d.ts.map +1 -1
- package/dist/runtime/async.js +4 -1
- package/dist/runtime/async.js.map +1 -1
- package/dist/runtime/auth.d.ts.map +1 -1
- package/dist/runtime/auth.js +12 -16
- package/dist/runtime/auth.js.map +1 -1
- package/dist/runtime/checks.d.ts.map +1 -1
- package/dist/runtime/checks.js +195 -218
- package/dist/runtime/checks.js.map +1 -1
- package/dist/runtime/finding.d.ts.map +1 -1
- package/dist/runtime/finding.js.map +1 -1
- package/dist/runtime/fixture.d.ts.map +1 -1
- package/dist/runtime/fixture.js.map +1 -1
- package/dist/runtime/progress.js +4 -4
- package/dist/runtime/progress.js.map +1 -1
- package/dist/runtime/run.d.ts.map +1 -1
- package/dist/runtime/run.js +6 -3
- package/dist/runtime/run.js.map +1 -1
- package/dist/runtime/validate.d.ts.map +1 -1
- package/dist/runtime/validate.js.map +1 -1
- package/dist/runtime/world.d.ts.map +1 -1
- package/dist/runtime/world.js.map +1 -1
- package/dist/spec/collection.d.ts.map +1 -1
- package/dist/spec/collection.js +2 -5
- package/dist/spec/collection.js.map +1 -1
- package/dist/spec/conventions.d.ts.map +1 -1
- package/dist/spec/conventions.js +2 -2
- package/dist/spec/conventions.js.map +1 -1
- package/dist/spec/extensions.d.ts.map +1 -1
- package/dist/spec/extensions.js +2 -6
- package/dist/spec/extensions.js.map +1 -1
- package/dist/spec/graph.d.ts.map +1 -1
- package/dist/spec/graph.js +2 -3
- package/dist/spec/graph.js.map +1 -1
- package/dist/spec/load.d.ts.map +1 -1
- package/dist/spec/load.js +1 -3
- package/dist/spec/load.js.map +1 -1
- package/labs/README.md +63 -63
- package/labs/annotated-openapi.yaml +5 -5
- package/labs/demo.mjs +2 -10
- package/package.json +84 -78
package/README.md
CHANGED
|
@@ -1,28 +1,70 @@
|
|
|
1
1
|
# oat
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/@lovrozagar/oat)
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
```bash
|
|
6
|
+
npm i -D @lovrozagar/oat
|
|
7
|
+
```
|
|
8
|
+
|
|
9
|
+
OpenAPI Tester — live **matrix testing** of a backend against its own OpenAPI document.
|
|
10
|
+
|
|
11
|
+
It reads the spec, talks to the running API, and treats every way to see a record as a cell in a matrix. Then it checks that those cells agree. It does not read your source, assume your framework, or hardcode a route.
|
|
6
12
|
|
|
7
|
-
-
|
|
8
|
-
|
|
9
|
-
-
|
|
13
|
+
A single-response check asks _did this JSON match its schema?_ oat asks that on every request it sends — generated bodies, 4xx probes, 500s, documented statuses. Most production bugs still pass that test:
|
|
14
|
+
|
|
15
|
+
- the row is on `GET /tables/{id}` and missing from `GET /tables`
|
|
16
|
+
- `?filter=status.eq.nope` returns every row (the backend dropped the param)
|
|
17
|
+
- `limit=2` yields 9 rows; `limit=100` yields 10 (the sort has no total order)
|
|
10
18
|
- `PATCH { name }` also cleared `instruction`
|
|
11
19
|
- `?filter=id.eq.<another tenant's id>` returns the row
|
|
12
20
|
|
|
13
|
-
|
|
21
|
+
Those are disagreements between **projections of the same fact**. That is the matrix.
|
|
22
|
+
|
|
23
|
+
**How the matrix is built.** oat inverts `x-invalidate` (or path heuristics) into an entity graph. Each entity gets a _read surface_: collection, item, filter, sort, page, cursor, select, search, parent routes, other tenants. It seeds a discriminating cohort (values whose lexical and numeric order disagree, LIKE metacharacters, unicode, nulls). Then it walks:
|
|
24
|
+
|
|
25
|
+
- **foundations** — create landed, the page walk covers the set, equality selects one, sort actually sorts
|
|
26
|
+
- **composition** — filter+sort, filter+select, search+filter, the triples; a filter must apply to the _collection_, not to the current page
|
|
27
|
+
- **writes** — PATCH is minimal, immutable fields stay put, two PATCHes do not clobber, replay does not duplicate
|
|
28
|
+
- **isolation** — a second principal with different `roots`, a same-tenant rank lattice, an invite that grants and then revokes
|
|
29
|
+
- **spec as adversary** — every field you _declared_ filterable / sortable / selectable actually is
|
|
30
|
+
|
|
31
|
+
There is no ground-truth database. A filter and its negation must partition the set. A page walk must cover the collection without gaps or dupes. List, item, and `id.eq.` must show the same field. One root cause is one finding; checks that depend on a broken primitive are `BLOCKED`, not a page of copies.
|
|
32
|
+
|
|
33
|
+
This file is the operator manual. An agent that has read it can install oat, write every kind of config, run every command, tag a document, interpret every outcome, and know what each check needs and asserts.
|
|
14
34
|
|
|
15
35
|
## Table of contents
|
|
16
36
|
|
|
17
37
|
- [Install](#install)
|
|
38
|
+
- [How a run works](#how-a-run-works)
|
|
18
39
|
- [Quick start](#quick-start)
|
|
40
|
+
- [Complete configs](#complete-configs)
|
|
19
41
|
- [Commands](#commands)
|
|
42
|
+
- [oat run](#oat-run)
|
|
43
|
+
- [oat doctor](#oat-doctor)
|
|
44
|
+
- [oat plan](#oat-plan)
|
|
45
|
+
- [oat serve](#oat-serve)
|
|
46
|
+
- [oat conformance](#oat-conformance)
|
|
20
47
|
- [Configuration](#configuration)
|
|
48
|
+
- [Top-level fields](#top-level-fields)
|
|
49
|
+
- [Principals](#principals)
|
|
50
|
+
- [Auth flows](#auth-flows)
|
|
51
|
+
- [Hooks](#hooks)
|
|
52
|
+
- [Environment interpolation](#environment-interpolation)
|
|
53
|
+
- [Loading TypeScript configs](#loading-typescript-configs)
|
|
54
|
+
- [How the model is derived](#how-the-model-is-derived)
|
|
55
|
+
- [Seeding](#seeding)
|
|
56
|
+
- [Query roles and grammars](#query-roles-and-grammars)
|
|
57
|
+
- [Pagination and envelopes](#pagination-and-envelopes)
|
|
21
58
|
- [OpenAPI meta tags](#openapi-meta-tags)
|
|
22
|
-
- [
|
|
59
|
+
- [Checks](#checks)
|
|
60
|
+
- [Verdicts, skips, and exit codes](#verdicts-skips-and-exit-codes)
|
|
23
61
|
- [Reports](#reports)
|
|
24
|
-
- [
|
|
25
|
-
- [
|
|
62
|
+
- [Progress logs](#progress-logs)
|
|
63
|
+
- [Programmatic API](#programmatic-api)
|
|
64
|
+
- [CI](#ci)
|
|
65
|
+
- [Reference defects (`oat serve --defects`)](#reference-defects-oat-serve---defects)
|
|
66
|
+
- [Limits and non-features](#limits-and-non-features)
|
|
67
|
+
- [Compared to schema fuzzers](#compared-to-schema-fuzzers)
|
|
26
68
|
- [Labs](#labs)
|
|
27
69
|
- [License](#license)
|
|
28
70
|
|
|
@@ -32,34 +74,56 @@ oat asserts the same fact through every projection the API offers, then compares
|
|
|
32
74
|
npm i -D @lovrozagar/oat
|
|
33
75
|
```
|
|
34
76
|
|
|
35
|
-
Requires Node.js 20
|
|
77
|
+
Requires **Node.js 20+**. The published CLI is compiled JavaScript; `npx oat` / `./node_modules/.bin/oat` is the entry.
|
|
78
|
+
|
|
79
|
+
The unscoped name `oat` on npm is a different project. Always install `@lovrozagar/oat`. The binary on PATH is still `oat`.
|
|
80
|
+
|
|
81
|
+
SQLite conformance (`npm test`, `oat conformance` with the sqlite backend) needs `node --experimental-sqlite` on Node 22. The published `oat` binary does not pass that flag for you; `npm test` in this repo does.
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
oat help # same as oat --help
|
|
85
|
+
oat --help
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Unknown commands and missing required flags exit `2`.
|
|
89
|
+
|
|
90
|
+
## How a run works
|
|
91
|
+
|
|
92
|
+
1. **Load** the OpenAPI document (URL or path). Internal `$ref`s are inlined. External `$ref`s are reported, never fetched.
|
|
93
|
+
2. **Model** entities by inverting `x-invalidate` (or path heuristics) into a read surface per entity.
|
|
94
|
+
3. **Authenticate** every configured principal (static headers and/or an auth flow). Credentials refresh themselves before they expire.
|
|
95
|
+
4. **Seed** a cohort of records per entity, in parent-before-child order, using each entity's create operation.
|
|
96
|
+
5. **Test** the matrix, one entity at a time: foundations first, then composition, writes, isolation, declared effects. Checks inside an entity stay ordered. Entities may run in parallel (`concurrency`).
|
|
97
|
+
6. **Teardown** everything the run created, unless `--keep-fixtures` / `keepFixtures: true`.
|
|
98
|
+
|
|
99
|
+
The first principal is the writer. Isolation needs a second principal with different `roots`. A rank lattice needs two or more principals that share `roots` and differ in `rank`. Invite checks need `x-invite` plus a peer with `inviteAs`.
|
|
100
|
+
|
|
101
|
+
oat never needs ground truth about your data. A filter and its negation must partition the set; a page walk must cover the collection; a record read four ways must read the same.
|
|
102
|
+
|
|
103
|
+
oat does **not** use OpenAPI `security` / `securitySchemes`, `servers[]`, cookies, webhooks, callbacks, or `links`. Auth is the config. The origin is `baseUrl`. Request bodies it sends are JSON.
|
|
36
104
|
|
|
37
105
|
## Quick start
|
|
38
106
|
|
|
39
|
-
|
|
107
|
+
The package ships a demo API (the same reference backend the self-test uses):
|
|
40
108
|
|
|
41
109
|
```bash
|
|
42
|
-
# terminal 1
|
|
110
|
+
# terminal 1 — prints a url, spec, and demo keys
|
|
43
111
|
oat serve --defects STALE_LIST,PATCH_REPLACES
|
|
44
112
|
|
|
45
113
|
# terminal 2
|
|
46
|
-
oat run --config labs/local.config.ts --base-url <url
|
|
47
|
-
```
|
|
48
|
-
|
|
114
|
+
oat run --config node_modules/@lovrozagar/oat/labs/local.config.ts --base-url <url from serve>
|
|
49
115
|
```
|
|
50
|
-
25 checks · 3 entities · 164 requests · 0.2s
|
|
51
116
|
|
|
52
|
-
|
|
53
|
-
table list projection does not reflect a completed write
|
|
54
|
-
table PATCH changed fields the request did not mention
|
|
117
|
+
Inside this repository (after `npm run build`):
|
|
55
118
|
|
|
56
|
-
|
|
57
|
-
|
|
119
|
+
```bash
|
|
120
|
+
oat serve --defects STALE_LIST,PATCH_REPLACES
|
|
121
|
+
oat run --config labs/local.config.ts --base-url <url>
|
|
58
122
|
```
|
|
59
123
|
|
|
60
|
-
`oat
|
|
124
|
+
`oat serve` with no `--defects` is a correct backend. The suite should report nothing.
|
|
61
125
|
|
|
62
|
-
Against your
|
|
126
|
+
Against your API:
|
|
63
127
|
|
|
64
128
|
```bash
|
|
65
129
|
oat doctor --spec https://api.example.com/openapi.json
|
|
@@ -67,93 +131,722 @@ oat plan --spec https://api.example.com/openapi.json
|
|
|
67
131
|
oat run --config oat.config.ts
|
|
68
132
|
```
|
|
69
133
|
|
|
70
|
-
`doctor` is the adoption command. It runs offline against the spec alone and reports every coverage gap, naming the
|
|
134
|
+
`doctor` is the adoption command. It runs offline against the spec alone and reports every coverage gap, naming the tag that would close it.
|
|
135
|
+
|
|
136
|
+
## Complete configs
|
|
137
|
+
|
|
138
|
+
These are copy-paste starting points. Real configs import `defineConfig` from `@lovrozagar/oat`. Files inside this repository import from `../dist/index.js` because they live in the source tree.
|
|
139
|
+
|
|
140
|
+
### Static API keys, two tenants (smallest useful)
|
|
141
|
+
|
|
142
|
+
```ts
|
|
143
|
+
// oat.config.ts
|
|
144
|
+
import { defineConfig } from "@lovrozagar/oat"
|
|
145
|
+
|
|
146
|
+
export default defineConfig({
|
|
147
|
+
spec: "https://api.example.com/openapi.json",
|
|
148
|
+
baseUrl: "https://api.example.com",
|
|
149
|
+
principals: [
|
|
150
|
+
{
|
|
151
|
+
id: "alpha",
|
|
152
|
+
headers: { authorization: "Bearer ${API_TOKEN}" },
|
|
153
|
+
roots: { project_id: "${PROJECT_A}" },
|
|
154
|
+
},
|
|
155
|
+
{
|
|
156
|
+
id: "beta",
|
|
157
|
+
headers: { authorization: "Bearer ${API_TOKEN_B}" },
|
|
158
|
+
roots: { project_id: "${PROJECT_B}" },
|
|
159
|
+
},
|
|
160
|
+
],
|
|
161
|
+
})
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
export API_TOKEN=… API_TOKEN_B=… PROJECT_A=proj_a PROJECT_B=proj_b
|
|
166
|
+
oat run --config oat.config.ts
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
One principal is enough to seed and run CRUD / query / schema checks. The second principal, with different `roots`, is what makes `tenant.*` run. Without it those checks are **did not apply**, not a pass.
|
|
170
|
+
|
|
171
|
+
Shipped as `labs/minimal.config.ts` (and in the npm package).
|
|
172
|
+
|
|
173
|
+
### JSON config
|
|
174
|
+
|
|
175
|
+
Same object. `${NAME}` is interpolated after load. There is no `defineConfig` wrapper.
|
|
176
|
+
|
|
177
|
+
```json
|
|
178
|
+
{
|
|
179
|
+
"spec": "https://api.example.com/openapi.json",
|
|
180
|
+
"baseUrl": "https://api.example.com",
|
|
181
|
+
"principals": [
|
|
182
|
+
{
|
|
183
|
+
"id": "alpha",
|
|
184
|
+
"headers": { "authorization": "Bearer ${API_TOKEN}" },
|
|
185
|
+
"roots": { "project_id": "${PROJECT_A}" }
|
|
186
|
+
}
|
|
187
|
+
],
|
|
188
|
+
"seed": 42,
|
|
189
|
+
"cohortSize": 7,
|
|
190
|
+
"concurrency": 1,
|
|
191
|
+
"outDir": "./oat-out"
|
|
192
|
+
}
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
```bash
|
|
196
|
+
oat run --config oat.config.json
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
### Demo server (operation-id login)
|
|
200
|
+
|
|
201
|
+
Shipped as `labs/local.config.ts`. Points at `oat serve`.
|
|
202
|
+
|
|
203
|
+
```ts
|
|
204
|
+
import { defineConfig } from "@lovrozagar/oat"
|
|
205
|
+
|
|
206
|
+
export default defineConfig({
|
|
207
|
+
spec: "/v1/openapi/spec",
|
|
208
|
+
baseUrl: "http://127.0.0.1:8787",
|
|
209
|
+
seed: 42,
|
|
210
|
+
principals: [
|
|
211
|
+
{
|
|
212
|
+
id: "alpha",
|
|
213
|
+
roots: { project_id: "proj_alpha" },
|
|
214
|
+
auth: {
|
|
215
|
+
credentialFrom: "$.access_token",
|
|
216
|
+
steps: [{ operationId: "auth.token", body: { key: "key_alpha" } }],
|
|
217
|
+
},
|
|
218
|
+
},
|
|
219
|
+
{
|
|
220
|
+
id: "beta",
|
|
221
|
+
roots: { project_id: "proj_beta" },
|
|
222
|
+
auth: {
|
|
223
|
+
credentialFrom: "$.access_token",
|
|
224
|
+
steps: [{ operationId: "auth.token", body: { key: "key_beta" } }],
|
|
225
|
+
},
|
|
226
|
+
},
|
|
227
|
+
],
|
|
228
|
+
})
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
`spec: "/v1/openapi/spec"` is resolved against `baseUrl`. `--base-url` on the CLI overrides the origin without editing the file.
|
|
232
|
+
|
|
233
|
+
### Two tenants plus a same-tenant rank lattice
|
|
234
|
+
|
|
235
|
+
See [Principals](#principals). Isolation keys off `roots`. Rank keys off `rank` with shared `roots`.
|
|
71
236
|
|
|
72
237
|
## Commands
|
|
73
238
|
|
|
74
239
|
```
|
|
75
|
-
oat run
|
|
76
|
-
oat
|
|
77
|
-
oat
|
|
78
|
-
oat serve
|
|
79
|
-
oat conformance
|
|
240
|
+
oat run --config <file> test a live backend and write a report
|
|
241
|
+
oat doctor --spec <url|file> what oat can and cannot test, and why
|
|
242
|
+
oat plan --spec <url|file> print the derived model (offline)
|
|
243
|
+
oat serve [--defects A,B] run the demo API
|
|
244
|
+
oat conformance self-test: injected defects vs detection
|
|
245
|
+
oat help
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
`--spec` for `doctor` / `plan` can be replaced by `--config` (the spec is read from the config). `--json` makes those two commands emit machine-readable output. `--base-url` on `doctor` / `plan` is only used to resolve a relative spec path.
|
|
249
|
+
|
|
250
|
+
`--untagged` is a **serve** flag (and a conformance concern). It is not a `run` flag.
|
|
251
|
+
|
|
252
|
+
### `oat run`
|
|
253
|
+
|
|
254
|
+
Requires `--config`. CLI flags override the same field in the config when both are set.
|
|
255
|
+
|
|
256
|
+
| flag | default | meaning |
|
|
257
|
+
| ----------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
258
|
+
| `--config` | required | module or JSON file, default export |
|
|
259
|
+
| `--base-url` | `config.baseUrl` | backend origin |
|
|
260
|
+
| `--only` | `config.only` or all entities | comma-separated entity names as `oat plan` prints them (singularised) |
|
|
261
|
+
| `--seed` | `config.seed` or `1` | fixture generation seed (reproducible) |
|
|
262
|
+
| `--out` | `config.outDir` or `./oat-out` | report directory |
|
|
263
|
+
| `--concurrency` | `config.concurrency` or `1` | entities in parallel. Use `1` on nested graphs; higher values insert children while a parent page-walk is running and invent pagination bugs |
|
|
264
|
+
| `--max-in-flight` | `config.maxInFlight` or `4` | HTTP requests allowed at once |
|
|
265
|
+
| `--keep-fixtures` | `config.keepFixtures` or false | do not DELETE what the run created |
|
|
266
|
+
| `--quiet` | false | no stderr progress; files under `--out` still update |
|
|
267
|
+
|
|
268
|
+
**Exit codes:** `0` no defects, `1` at least one root-cause finding (`BACKEND_BUG`, `SPEC_BUG`, `SECURITY`, `AMBIGUITY`), `2` usage error (missing `--config`, no principals). `COVERAGE_GAP` and `BLOCKED` do not fail the process.
|
|
269
|
+
|
|
270
|
+
Example:
|
|
271
|
+
|
|
272
|
+
```bash
|
|
273
|
+
oat run --config oat.config.ts --only store,product --concurrency 1 --out oat-out/prod
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
`--only store,product` matches the **entity names** from `oat plan`, not path segments. `/v1/stores` is usually the entity `store`. If a name is unknown, that entity is simply not tested (the others still run).
|
|
277
|
+
|
|
278
|
+
A run with no principals exits `2`. Isolation checks then need a second principal; they are skipped, not failed, when only one is present.
|
|
279
|
+
|
|
280
|
+
### `oat doctor`
|
|
281
|
+
|
|
282
|
+
Offline. Loads the document, builds the model, prints coverage.
|
|
283
|
+
|
|
284
|
+
```bash
|
|
285
|
+
oat doctor --spec https://api.example.com/openapi.json
|
|
286
|
+
oat doctor --config oat.config.ts --json
|
|
287
|
+
oat doctor --spec ./openapi.yaml --base-url https://api.example.com
|
|
80
288
|
```
|
|
81
289
|
|
|
82
|
-
|
|
290
|
+
Human output:
|
|
291
|
+
|
|
292
|
+
- `trackable` — entities with an identity and a read surface
|
|
293
|
+
- `listable` — those that also have a list (query checks need this)
|
|
294
|
+
- tags that are absent, and the checks each tag would unlock
|
|
295
|
+
- tags that would **sharpen** checks that already run (`x-query`, `x-tenant`)
|
|
296
|
+
- per-operation gaps (`x-entity` could not be inferred, assumed tenant param, …)
|
|
297
|
+
- external `$ref`s that were not fetched
|
|
83
298
|
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
299
|
+
`--json` shape:
|
|
300
|
+
|
|
301
|
+
```json
|
|
302
|
+
{
|
|
303
|
+
"blocking": 1,
|
|
304
|
+
"entities": 12,
|
|
305
|
+
"trackableEntities": 10,
|
|
306
|
+
"testableEntities": 10,
|
|
307
|
+
"listableEntities": 8,
|
|
308
|
+
"roots": ["organization_id"],
|
|
309
|
+
"externalRefs": ["https://example.com/shared.yaml"],
|
|
310
|
+
"gaps": [{ "operationId": "table.list", "tag": "x-query", "detail": "…" }]
|
|
311
|
+
}
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
Exit `1` if there are **blocking** gaps: entities that are not trackable (no identity / no read), or the document has roots oat cannot create. Advisory gaps (missing `x-query`, no `x-async`) print and still exit `0`.
|
|
315
|
+
|
|
316
|
+
### `oat plan`
|
|
317
|
+
|
|
318
|
+
Offline. Prints the derived entity graph, operations, and query capability.
|
|
319
|
+
|
|
320
|
+
```bash
|
|
321
|
+
oat plan --spec ./openapi.yaml
|
|
322
|
+
oat plan --config oat.config.ts --json
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
Human columns:
|
|
326
|
+
|
|
327
|
+
```
|
|
328
|
+
entity CLRUD ident read surface
|
|
329
|
+
store CLRU· id 2 route(s) (inferred)
|
|
330
|
+
GET /v1/stores
|
|
331
|
+
GET /v1/stores/{store_id}
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
`CLRUD` is Create / List / Read / Update / Delete. `·` means that slot is missing. `ident` is the identity property. Read surface is declared (`x-invalidate`) or inferred (sibling collection/item routes).
|
|
335
|
+
|
|
336
|
+
`--json` is `{ entities, operations, roots }` — the full `SpecModel` maps, including conventions, query capability, async, invite, and path params. Use this when you need to know what oat will call something.
|
|
337
|
+
|
|
338
|
+
### `oat serve`
|
|
339
|
+
|
|
340
|
+
In-process demo API. Same fixture as conformance.
|
|
341
|
+
|
|
342
|
+
```bash
|
|
343
|
+
oat serve
|
|
344
|
+
oat serve --defects STALE_LIST,PATCH_REPLACES
|
|
345
|
+
oat serve --backend sqlite --dialect classic
|
|
346
|
+
oat serve --untagged
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
| flag | default | meaning |
|
|
350
|
+
| ------------ | ----------- | -------------------------------------------------------------------------------------- |
|
|
351
|
+
| `--backend` | `memory` | `memory` \| `sqlite` \| `postgres` |
|
|
352
|
+
| `--dialect` | `postgrest` | `postgrest` \| `classic` \| `linked` \| `jsonapi` \| `plain` |
|
|
353
|
+
| `--defects` | none | comma-separated names from [Reference defects](#reference-defects-oat-serve---defects) |
|
|
354
|
+
| `--untagged` | false | serve the same API behind a spec with every `x-*` tag stripped |
|
|
355
|
+
|
|
356
|
+
Printed keys: `key_alpha` (tenant `proj_alpha`), `key_beta` (tenant `proj_beta`). Spec: `{url}/v1/openapi/spec`. Stop with ctrl-c.
|
|
357
|
+
|
|
358
|
+
`labs/local.config.ts` is written for this server.
|
|
359
|
+
|
|
360
|
+
Dialects are **reference-backend shapes**, not something you configure against your API. They exist so conformance proves checks read the document rather than one fixture's spelling:
|
|
361
|
+
|
|
362
|
+
| dialect | filter | sort | select | page model | envelope |
|
|
363
|
+
| ----------- | --------------------------- | ---------- | ---------------- | ------------------- | --------------------------------- |
|
|
364
|
+
| `postgrest` | `filter=status.eq.active` | `name.asc` | `select=id,name` | `page` + `cursor` | entity-named + `count`/`hasMore` |
|
|
365
|
+
| `classic` | `filter=status=eq:active` | `sort=` | `fields=` | `page` + `per_page` | `{ data, total_count, has_more }` |
|
|
366
|
+
| `linked` | postgrest | dotted | `fields=` | `offset` + `limit` | raw array + `Link: rel=next` |
|
|
367
|
+
| `jsonapi` | postgrest | `-name` | `fields[table]=` | `page` + `size` | `{ data, total, has_more }` |
|
|
368
|
+
| `plain` | `?status=active` (equality) | `name:asc` | `fields=` | `page` + `limit` | `{ items, total, has_more }` |
|
|
369
|
+
|
|
370
|
+
`postgres` needs a server on the default `postgres` database (local, default `postgres` driver connection). `sqlite` needs Node's `node:sqlite` (`--experimental-sqlite` on Node 22). Missing backends fail at serve time rather than falling back.
|
|
371
|
+
|
|
372
|
+
### `oat conformance`
|
|
373
|
+
|
|
374
|
+
Self-test. Not for your API. Injects named defects into the reference backend and asserts oat reports the matching check.
|
|
375
|
+
|
|
376
|
+
```bash
|
|
377
|
+
# this repo
|
|
378
|
+
npm test
|
|
379
|
+
|
|
380
|
+
# after install, from a checkout with --experimental-sqlite if you want sqlite
|
|
381
|
+
oat conformance
|
|
382
|
+
oat conformance --backend memory --dialect plain
|
|
383
|
+
oat conformance --fuzz 300 --max-defects 12 --seed 7
|
|
384
|
+
oat conformance --precision 60 --backend memory
|
|
385
|
+
oat conformance --parser
|
|
386
|
+
oat conformance --backend d1
|
|
387
|
+
oat conformance --only STALE_LIST,PATCH_REPLACES
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
| flag | meaning |
|
|
391
|
+
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
|
|
392
|
+
| `--backend` | `memory` \| `sqlite` \| `postgres` \| `d1`. Default: every _local_ backend that is available. `d1` is never default — it is remote |
|
|
393
|
+
| `--dialect` | pin one shape; default runs postgrest on each backend plus classic/linked/jsonapi/plain on memory |
|
|
394
|
+
| `--fuzz [n]` | random _sets_ of defects (default 25 if flag is bare) |
|
|
395
|
+
| `--max-defects` | cap per fuzz combination (default 4) |
|
|
396
|
+
| `--precision [n]` | vary _data_ against a correct backend; any finding is a false positive (default 50 if flag is bare) |
|
|
397
|
+
| `--seed` | replay a fuzz/precision run |
|
|
398
|
+
| `--parser` | only the hostile-document + example-spec + tag-unlock suites |
|
|
399
|
+
| `--only` | restrict injected defects (comma-separated `STALE_LIST,…`) |
|
|
400
|
+
|
|
401
|
+
A default `oat conformance` (no `--fuzz` / `--precision` / `--parser`) also runs a 40-case combination smoke on memory after the one-at-a-time matrix.
|
|
402
|
+
|
|
403
|
+
D1 needs `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_D1_DATABASE_ID`, `CLOUDFLARE_API_TOKEN`. Postgres needs a reachable server on the default connection (`database: "postgres"`). Missing backends are skipped with a printed reason, not treated as a pass.
|
|
404
|
+
|
|
405
|
+
`--parser` still always runs first (hostile documents, `labs/annotated-openapi.yaml` model lock, tag-unlock map). Exit `1` if any parser, matrix, fuzz, or precision case fails.
|
|
98
406
|
|
|
99
407
|
## Configuration
|
|
100
408
|
|
|
101
|
-
Two inputs, always: the spec
|
|
409
|
+
Two inputs, always: **the spec** and **a config file**. Backend-specific knowledge lives in `x-*` tags and this file. A backend adopts oat by adding tags, not by adapting to oat.
|
|
410
|
+
|
|
411
|
+
A config is:
|
|
102
412
|
|
|
103
|
-
-
|
|
104
|
-
-
|
|
413
|
+
- `.ts` / `.js` / `.mjs` with `export default defineConfig({ ... })`, or
|
|
414
|
+
- `.json` with the same object.
|
|
105
415
|
|
|
106
|
-
|
|
416
|
+
Named export without `default` is also accepted (`module.default ?? module`).
|
|
107
417
|
|
|
108
|
-
|
|
418
|
+
### Top-level fields
|
|
109
419
|
|
|
110
420
|
```ts
|
|
111
421
|
import { defineConfig } from "@lovrozagar/oat"
|
|
112
422
|
|
|
113
423
|
export default defineConfig({
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
],
|
|
424
|
+
spec: "https://api.example.com/openapi.json", // URL or filesystem path; JSON or YAML
|
|
425
|
+
baseUrl: "https://api.example.com",
|
|
426
|
+
principals: [/* at least one; see below */],
|
|
427
|
+
hooks: {/* optional */},
|
|
428
|
+
globalHeaders: { "x-request-id": "oat" }, // sent on every request; oat does not inspect them
|
|
429
|
+
roots: { org_id: "org_shared" }, // path params oat cannot create; also declarable via x-root
|
|
430
|
+
seed: 42, // fixture generation; a failing run with the same seed is identical
|
|
431
|
+
cohortSize: 12, // records created per entity (default 7)
|
|
432
|
+
concurrency: 1, // entities in parallel
|
|
433
|
+
maxInFlight: 4, // HTTP in flight
|
|
434
|
+
only: ["store", "product"], // restrict entities
|
|
435
|
+
keepFixtures: false,
|
|
436
|
+
outDir: "./oat-out",
|
|
128
437
|
})
|
|
129
438
|
```
|
|
130
439
|
|
|
131
|
-
|
|
440
|
+
| field | required | default | notes |
|
|
441
|
+
| --------------- | -------- | ----------- | --------------------------------------------------------------- |
|
|
442
|
+
| `spec` | yes | | See [Spec loading](#spec-loading) |
|
|
443
|
+
| `baseUrl` | yes | | Origin. OpenAPI `servers[]` is ignored |
|
|
444
|
+
| `principals` | yes | | Non-empty. First is the writer |
|
|
445
|
+
| `hooks` | no | | `resolveOutOfBand`, `teardownPrincipal` |
|
|
446
|
+
| `globalHeaders` | no | `{}` | Merged under per-request headers. Opaque to oat |
|
|
447
|
+
| `roots` | no | `{}` | Shared path params (merged with each principal's `roots`) |
|
|
448
|
+
| `seed` | no | `1` | Integer. Same seed → same fixture bodies |
|
|
449
|
+
| `cohortSize` | no | `7` | Sliced from the 7 built-in variants. Larger repeats the pattern |
|
|
450
|
+
| `concurrency` | no | `1` | Entity-level only. Checks inside one entity stay ordered |
|
|
451
|
+
| `maxInFlight` | no | `4` | Across the whole run |
|
|
452
|
+
| `only` | no | all | Entity names from `oat plan` |
|
|
453
|
+
| `keepFixtures` | no | `false` | Skip DELETE at the end |
|
|
454
|
+
| `outDir` | no | `./oat-out` | Created if missing |
|
|
455
|
+
|
|
456
|
+
`spec` may be a path relative to `baseUrl` (`/v1/openapi/spec`) or an absolute URL or a file.
|
|
457
|
+
|
|
458
|
+
CLI `--base-url`, `--only`, `--seed`, `--out`, `--concurrency`, `--max-in-flight`, `--keep-fixtures` override these when passed.
|
|
459
|
+
|
|
460
|
+
### Spec loading
|
|
461
|
+
|
|
462
|
+
Resolved in this order, never by guessing the string's "look":
|
|
463
|
+
|
|
464
|
+
1. Absolute `http(s)://` or `file://` — used as given.
|
|
465
|
+
2. A path that exists on disk, relative to the working directory.
|
|
466
|
+
3. Anything else, when `baseUrl` is known — resolved against it (`/v1/openapi/spec`, `openapi.json`).
|
|
467
|
+
|
|
468
|
+
JSON if the first non-space character is `{` or `[`, otherwise YAML. Empty files error. A JSON document with more opening than closing brackets is diagnosed as truncated (proxy / download limit), not as a syntax error.
|
|
469
|
+
|
|
470
|
+
OpenAPI 3.0 and 3.1 both work. oat reads `paths`, operations, parameters, request/response JSON schemas, and `x-*` extensions. It does not require a particular `openapi:` version string.
|
|
471
|
+
|
|
472
|
+
Internal `$ref`s are dereferenced. External `$ref`s stay unresolved and show up in `oat doctor` / the JSON `externalRefs` list.
|
|
473
|
+
|
|
474
|
+
### Principals
|
|
132
475
|
|
|
133
476
|
```ts
|
|
134
477
|
{
|
|
135
|
-
id: "alpha",
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
478
|
+
id: "alpha", // required, stable name in reports
|
|
479
|
+
headers: { authorization: "Bearer …" }, // static; enough for a long-lived key
|
|
480
|
+
auth: { /* AuthFlow — see below */ },
|
|
481
|
+
roots: { org_id: "org_alpha" }, // this principal's tenant / path params
|
|
482
|
+
rootsFromFlow: { org_id: "orgId" }, // take path params from values the auth flow bound
|
|
483
|
+
role: "owner", // free-form label in reports
|
|
484
|
+
rank: 2, // higher can do everything a lower rank can; default 0
|
|
485
|
+
inviteAs: "key_beta", // how an owner names this principal in an invite body
|
|
486
|
+
}
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
Rules that matter:
|
|
490
|
+
|
|
491
|
+
- **Isolation** (`tenant.*`) needs two principals whose `roots` differ.
|
|
492
|
+
- **Rank** (`auth.rank-is-monotonic`) needs two principals with the _same_ `roots` and different `rank`.
|
|
493
|
+
- **Invite** (`auth.invite-grants-then-revokes`) needs `x-invite` on the spec and a _different-tenant_ principal with `inviteAs` set.
|
|
494
|
+
- Extra principals are not ignored. Isolation picks the first different-`roots` peer. Rank uses the same-tenant pair.
|
|
495
|
+
- `headers` and `auth` compose: static headers are sent, then the flow's credential header is merged on top.
|
|
496
|
+
- A principal with only `headers` (no `auth`) never hits a login route.
|
|
497
|
+
|
|
498
|
+
Example — two tenants plus a same-tenant lattice:
|
|
499
|
+
|
|
500
|
+
```ts
|
|
501
|
+
principals: [
|
|
502
|
+
{
|
|
503
|
+
id: "alpha",
|
|
504
|
+
role: "owner",
|
|
505
|
+
rank: 2,
|
|
506
|
+
auth: {
|
|
507
|
+
credentialFrom: "$.access_token",
|
|
508
|
+
steps: [{ operationId: "auth.token", body: { key: "key_alpha" } }],
|
|
509
|
+
},
|
|
510
|
+
roots: { org_id: "org_alpha" },
|
|
511
|
+
},
|
|
512
|
+
{
|
|
513
|
+
id: "alpha_member",
|
|
514
|
+
role: "member",
|
|
515
|
+
rank: 1,
|
|
516
|
+
auth: {
|
|
517
|
+
credentialFrom: "$.access_token",
|
|
518
|
+
steps: [{ operationId: "auth.token", body: { key: "key_alpha_member" } }],
|
|
519
|
+
},
|
|
520
|
+
roots: { org_id: "org_alpha" },
|
|
521
|
+
},
|
|
522
|
+
{
|
|
523
|
+
id: "beta",
|
|
524
|
+
role: "owner",
|
|
525
|
+
rank: 2,
|
|
526
|
+
inviteAs: "key_beta",
|
|
527
|
+
auth: {
|
|
528
|
+
credentialFrom: "$.access_token",
|
|
529
|
+
steps: [{ operationId: "auth.token", body: { key: "key_beta" } }],
|
|
530
|
+
},
|
|
531
|
+
roots: { org_id: "org_beta" },
|
|
532
|
+
},
|
|
533
|
+
]
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
### Auth flows
|
|
537
|
+
|
|
538
|
+
```ts
|
|
539
|
+
auth: {
|
|
540
|
+
steps: [ /* at least one */ ],
|
|
541
|
+
credentialFrom: "$.access_token", // JSON path in the last (or saved) response
|
|
542
|
+
expiresInFrom: "$.expires_in", // lifetime in seconds
|
|
543
|
+
header: "authorization", // default
|
|
544
|
+
template: "Bearer {credential}", // default
|
|
545
|
+
assumeTtlMs: 3600000, // used only if neither expiresInFrom nor JWT exp is present
|
|
546
|
+
}
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
Expiry is `expiresInFrom` (seconds) → JWT `exp` claim → `assumeTtlMs`. Refresh happens before dispatch, at roughly three quarters of the assumed window (`assumeTtlMs` default 300000 ms when used as the window), never by retrying a 401.
|
|
550
|
+
|
|
551
|
+
Each step is one of:
|
|
552
|
+
|
|
553
|
+
**Operation step** (prefer this — survives the path moving):
|
|
554
|
+
|
|
555
|
+
```ts
|
|
556
|
+
{
|
|
557
|
+
operationId: "auth.token",
|
|
558
|
+
body: { key: "${API_KEY}" },
|
|
559
|
+
headers: { "x-extra": "1" },
|
|
560
|
+
query: { realm: "test" },
|
|
561
|
+
saveAs: { token: "$.access_token" },
|
|
562
|
+
saveClaimsFrom: { token: "$.access_token", bind: { orgId: "orgs.0.oid" } },
|
|
563
|
+
bind: { address: "user@example.test" }, // literals, with {name} interpolation
|
|
564
|
+
expect: [200], // default: any 2xx
|
|
565
|
+
}
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
**Request step** (when the document has no auth operations):
|
|
569
|
+
|
|
570
|
+
```ts
|
|
571
|
+
{
|
|
572
|
+
method: "POST",
|
|
573
|
+
path: "/v1/auth/register/email",
|
|
574
|
+
body: { email: "{address}", password: "…" },
|
|
575
|
+
bind: { address: "oat-alpha@example.test" },
|
|
576
|
+
}
|
|
577
|
+
```
|
|
578
|
+
|
|
579
|
+
**Out-of-band step** (email link, OTP — oat cannot collect this itself):
|
|
580
|
+
|
|
581
|
+
```ts
|
|
582
|
+
{ outOfBand: { address: "{address}", kind: "email-verify", as: "verifyToken" } }
|
|
583
|
+
```
|
|
584
|
+
|
|
585
|
+
Later steps interpolate `{name}` from the flow scope. `saveAs` paths are `$.foo.bar` / `$.orgs.0.id` (dot + numeric index only; no JSON Pointer, no filters). `saveClaimsFrom` reads a JWT's _claims_ (signature is not verified — oat is reading its own credential). `rootsFromFlow` maps path parameter names to those bound keys.
|
|
586
|
+
|
|
587
|
+
`bind` on a step runs **before** the request. `saveAs` / `saveClaimsFrom` run **after**. `credentialFrom` is read from the last HTTP response unless a step already saved `credential`.
|
|
588
|
+
|
|
589
|
+
If a step's status is not acceptable, auth fails the run (not a finding): `oat: principal "alpha" failed at auth step 2 (POST /v1/…)`.
|
|
590
|
+
|
|
591
|
+
A complete register → verify → use-claims example:
|
|
592
|
+
|
|
593
|
+
```ts
|
|
594
|
+
function signUp(email: string): AuthFlow {
|
|
595
|
+
return {
|
|
596
|
+
credentialFrom: "$.access_token",
|
|
597
|
+
expiresInFrom: "$.access_token_expires_in",
|
|
598
|
+
steps: [
|
|
599
|
+
{
|
|
600
|
+
bind: { address: email },
|
|
601
|
+
body: { email, password: "…" },
|
|
602
|
+
method: "POST",
|
|
603
|
+
path: "/v1/auth/register/email",
|
|
604
|
+
},
|
|
605
|
+
{ outOfBand: { address: email, as: "verifyToken", kind: "email-verify" } },
|
|
606
|
+
{
|
|
607
|
+
body: { token: "{verifyToken}" },
|
|
608
|
+
method: "POST",
|
|
609
|
+
path: "/v1/auth/email/verify",
|
|
610
|
+
saveClaimsFrom: {
|
|
611
|
+
token: "$.access_token",
|
|
612
|
+
bind: { orgId: "orgs.0.oid", projectId: "orgs.0.pids.0" },
|
|
613
|
+
},
|
|
614
|
+
},
|
|
615
|
+
],
|
|
616
|
+
}
|
|
617
|
+
}
|
|
618
|
+
|
|
619
|
+
principals: [
|
|
620
|
+
{
|
|
621
|
+
id: "alpha",
|
|
622
|
+
auth: signUp("oat-alpha@example.test"),
|
|
623
|
+
rootsFromFlow: { organization_id: "orgId", project_id: "projectId" },
|
|
624
|
+
},
|
|
625
|
+
]
|
|
626
|
+
```
|
|
627
|
+
|
|
628
|
+
The address used for `teardownPrincipal` is `scope.address` or `scope.email` (set via `bind: { address }` or `bind: { email }`).
|
|
629
|
+
|
|
630
|
+
### Hooks
|
|
631
|
+
|
|
632
|
+
```ts
|
|
633
|
+
hooks: {
|
|
634
|
+
// Return null to retry (attempt is 1-based). oat backs off until a value arrives.
|
|
635
|
+
resolveOutOfBand: async ({ address, kind, attempt }) => {
|
|
636
|
+
const token = await readMailCatcher(address, kind)
|
|
637
|
+
return token // or null
|
|
638
|
+
},
|
|
639
|
+
// Remove a principal this run provisioned (and everything it created).
|
|
640
|
+
teardownPrincipal: async (address) => {
|
|
641
|
+
await fetch(`https://api.example.com/test/cleanup?email=${address}`, { method: "DELETE" })
|
|
139
642
|
},
|
|
140
|
-
roots: { org_id: "org_alpha" },
|
|
141
643
|
}
|
|
142
644
|
```
|
|
143
645
|
|
|
144
|
-
`resolveOutOfBand`
|
|
646
|
+
Without `resolveOutOfBand`, an `outOfBand` step cannot complete. oat polls the hook up to **6** times: 200 ms, then doubling, capped at 3000 ms. Returning `""` is treated like `null`.
|
|
647
|
+
|
|
648
|
+
Without `teardownPrincipal`, provisioned accounts are reported as leftover rather than cascade-deleted. Per-record DELETE still runs for seeded rows when a delete (or `x-cleanup`) exists.
|
|
649
|
+
|
|
650
|
+
### Environment interpolation
|
|
651
|
+
|
|
652
|
+
After the module loads, every string in the config is scanned for `${NAME}`:
|
|
653
|
+
|
|
654
|
+
```ts
|
|
655
|
+
headers: {
|
|
656
|
+
authorization: "Bearer ${API_TOKEN}"
|
|
657
|
+
}
|
|
658
|
+
```
|
|
659
|
+
|
|
660
|
+
If `API_TOKEN` is unset, oat exits with an error. Do not commit secrets; put them in the environment.
|
|
661
|
+
|
|
662
|
+
Template literals in a `.ts` config (`Bearer ${process.env.API_TOKEN}`) are evaluated by Node _before_ oat sees the object. Either style works; `${NAME}` is what a `.json` config can use.
|
|
663
|
+
|
|
664
|
+
Names match `[A-Z0-9_]+` case-insensitively.
|
|
665
|
+
|
|
666
|
+
### Loading TypeScript configs
|
|
667
|
+
|
|
668
|
+
`.js` / `.mjs` / `.json` load everywhere.
|
|
669
|
+
|
|
670
|
+
`.ts` configs require a runtime that can import TypeScript: Node 22.6+ with `--experimental-strip-types`, or Node 23+. The published `oat` binary is itself JS; it still has to `import()` your config. If that fails, the error says so. Workaround: compile the config, or write `.mjs`.
|
|
671
|
+
|
|
672
|
+
```bash
|
|
673
|
+
node --experimental-strip-types ./node_modules/@lovrozagar/oat/dist/cli.js run --config oat.config.ts
|
|
674
|
+
```
|
|
675
|
+
|
|
676
|
+
## How the model is derived
|
|
677
|
+
|
|
678
|
+
`oat plan` is this model. Checks never see raw paths; they see entities, actions, and query **roles**.
|
|
679
|
+
|
|
680
|
+
### Entity name and action
|
|
681
|
+
|
|
682
|
+
**Explicit:** `x-entity: { name, action, identity? }` on the operation.
|
|
683
|
+
|
|
684
|
+
**Heuristic** (when the tag is absent):
|
|
685
|
+
|
|
686
|
+
1. Split the path on `/`. Ignore `{param}` segments.
|
|
687
|
+
2. The last non-parameter segment is the noun. `v1` / `v2` / `vN` is never a noun.
|
|
688
|
+
3. Singularise that noun (`stores` → `store`, `batches` → `batch`). Irregulars include `people→person`, `categories→category`, `campuses→campus`, `statuses→status`, `children→child`, `companies→company`, `addresses→address`, `indices→index`, `queries→query`, `properties→property`, `entities→entity`, `inboxes→inbox`. Endings `us|ss|is|os|as|ics|ews|ess|ous|sis` are left alone (`status` stays `status`).
|
|
689
|
+
4. If a non-parameter segment follows the noun (`/rows/aggregate`, `/tables/{id}/restore`), action is `action`.
|
|
690
|
+
5. Otherwise: `GET` collection → `list`, `GET` item → `read`, `POST` collection → `create`, `POST` item → `action`, `PUT`/`PATCH` → `update`, `DELETE` → `delete`.
|
|
691
|
+
|
|
692
|
+
If no noun can be found, the operation is untracked and `doctor` records an `x-entity` gap.
|
|
693
|
+
|
|
694
|
+
`--only` and report entity names are these singular names.
|
|
695
|
+
|
|
696
|
+
### Identity
|
|
697
|
+
|
|
698
|
+
`x-entity.identity` wins. Else the first of `id`, `uuid`, `slug`, `key`, `name` that is **required** on the item schema, else the first of those that exists, else the trailing path-param suffix (`{table_id}` → `id`). Without an identity the entity is not trackable.
|
|
699
|
+
|
|
700
|
+
### Read surface
|
|
701
|
+
|
|
702
|
+
The set of `GET` routes through which an instance is visible.
|
|
703
|
+
|
|
704
|
+
- Declared: every `"METHOD /path"` in any `x-invalidate` that refers to this entity.
|
|
705
|
+
- Inferred: sibling collection and item routes on the same path prefix as a mutator.
|
|
706
|
+
|
|
707
|
+
`invalidation.declared-route-changes` only runs when a mutator's `x-invalidate` names **another** entity's route.
|
|
708
|
+
|
|
709
|
+
### Generated / immutable / soft-delete / tenant
|
|
710
|
+
|
|
711
|
+
See [OpenAPI meta tags](#openapi-meta-tags). Fallbacks:
|
|
712
|
+
|
|
713
|
+
- `readOnly: true` counts as generated (omitted from create bodies).
|
|
714
|
+
- No immutability testing without `x-immutable`.
|
|
715
|
+
- Tenant param: `x-tenant` or a path param matching `org|organization|tenant|workspace|account|project|app` + optional `_id`/`_slug`. Inferred tenants make a cross-tenant read `AMBIGUITY`, not `SECURITY`.
|
|
716
|
+
|
|
717
|
+
### Idempotency
|
|
718
|
+
|
|
719
|
+
No meta tag. If create declares a header matching `Idempotency-Key` / `Idempotence-Key` / `X-Idempotency-Key` (spaces ignored, case-insensitive), `idempotency.replay-does-not-duplicate` runs.
|
|
720
|
+
|
|
721
|
+
## Seeding
|
|
722
|
+
|
|
723
|
+
Per entity, oat POSTs the create body built from the request JSON schema.
|
|
724
|
+
|
|
725
|
+
Default cohort is **7** records, one of each variant, sliced by `cohortSize`:
|
|
726
|
+
|
|
727
|
+
| variant | what it is for |
|
|
728
|
+
| --------------- | -------------------------------------------------- |
|
|
729
|
+
| `baseline` | `"Quarterly Report N"` |
|
|
730
|
+
| `lexical-first` | sorts first (`"aaa first alphabetically"`) |
|
|
731
|
+
| `lexical-last` | sorts last (`"zzz last alphabetically"`) |
|
|
732
|
+
| `null-heavy` | `null` on every nullable field |
|
|
733
|
+
| `unicode` | `"日本語 café ñandú"` |
|
|
734
|
+
| `metacharacter` | `"100% _off_ *everything*"` — LIKE / escape probes |
|
|
735
|
+
| `boundary` | empty / maxLength / numeric `maximum` |
|
|
736
|
+
|
|
737
|
+
Numbers use the ladder `1, 2, 5, 10, 20, 50, 100` so **lexical order ≠ numeric order** (otherwise a TEXT compare looks correct). Enums walk `index % enum.length`. `readOnly` / `x-generated` fields are omitted. Required fields that cannot be generated get a type fallback (`0`, `false`, `[]`, `{}`, `"value"`). Arrays honour `minItems` (never send `[]` when `minItems ≥ 1`). Nested objects stop at depth 4.
|
|
738
|
+
|
|
739
|
+
Parent path parameters are created first (depth-first through the owning entity's create). Config / principal `roots` fill parameters oat cannot create.
|
|
740
|
+
|
|
741
|
+
A create that returns `>= 300` on the first variant fails the entity (downstream checks `BLOCKED`). Later variants that fail just shorten the cohort — a partial cohort is still used.
|
|
742
|
+
|
|
743
|
+
`--seed` / `seed` makes the bodies identical across runs. It does not make server-assigned ids identical.
|
|
744
|
+
|
|
745
|
+
Teardown DELETEs created rows (or the `x-cleanup` route) newest-first. Failures and missing delete routes are printed as leftovers, not as check findings. `keepFixtures: true` skips this.
|
|
746
|
+
|
|
747
|
+
## Query roles and grammars
|
|
748
|
+
|
|
749
|
+
Checks do not look for a parameter _named_ `filter`. They resolve **roles** from aliases, then write values in the grammar the document demonstrates.
|
|
750
|
+
|
|
751
|
+
| role | aliases (normalised: case, `_` / `-`, `perPage` → `per_page`) |
|
|
752
|
+
| ----------------- | ----------------------------------------------------------------------------------- |
|
|
753
|
+
| filter | `filter`, `where`, `query`, `conditions` |
|
|
754
|
+
| order | `order`, `order_by`, `sort`, `sort_by`, `ordering` |
|
|
755
|
+
| select | `select`, `fields`, `field`, `include_fields`, `projection`, `only` |
|
|
756
|
+
| search | `q`, `search`, `query_text`, `term`, `keyword`, `text` |
|
|
757
|
+
| limit (page size) | `limit`, `per_page`, `page_size`, `pagesize`, `count`, `max_results`, `top`, `size` |
|
|
758
|
+
| page | `page`, `page_number`, `pagenum`, `p` |
|
|
759
|
+
| offset | `offset`, `skip`, `start`, `from` |
|
|
760
|
+
| cursor | `cursor`, `after`, `starting_after`, `next`, `page_token`, `continuation` |
|
|
761
|
+
|
|
762
|
+
A bracketed suffix is a _value_ in the name (`fields[articles]`, `filter[status]`), not part of the role. `count` is a page size only when it looks like one (has `maximum` or a default); otherwise it is treated as a total.
|
|
763
|
+
|
|
764
|
+
A bounded integer with a default that matches no alias is still taken as page size. A 1-based integer with no maximum is taken as page number.
|
|
765
|
+
|
|
766
|
+
**Filter grammars** — how oat **writes** a term:
|
|
767
|
+
|
|
768
|
+
| name | `eq` / `neq` / `gt` / `like` example | `and` / `or` |
|
|
769
|
+
| ----------- | ------------------------------------------------------------------------- | ---------------------------------- |
|
|
770
|
+
| `postgrest` | `filter=status.eq.active`, `name.neq.x`, `price.gt.10`, `name.like.*foo*` | `and(a.eq.1,b.eq.2)`, `or(...)` |
|
|
771
|
+
| `colon` | `filter=status=eq:active` (comma-joined terms; no grouping) | not expressible; those checks skip |
|
|
772
|
+
| `equality` | `?status=active` (one query param per field). Only `eq` is expressible | not expressible |
|
|
773
|
+
|
|
774
|
+
Operators oat can emit: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `like`. Anything else a check needs that the grammar cannot write becomes **did not apply**, not a failed request.
|
|
775
|
+
|
|
776
|
+
**Sort grammars:** `name.asc` (dotted), `-name` (prefixed / JSON:API; ascending is the bare name), `name:asc` (colon), `name asc` (spaced).
|
|
777
|
+
|
|
778
|
+
**Select grammars:** `id,name` (csv) or `fields[table]=id,name` (bracketed). If the parameter is already named `fields[articles]`, that name is used verbatim.
|
|
779
|
+
|
|
780
|
+
### How a grammar is inferred
|
|
781
|
+
|
|
782
|
+
`x-query.grammar` wins when it is `postgrest` | `colon` | `equality`.
|
|
783
|
+
|
|
784
|
+
Otherwise oat concatenates the filter/order/select parameter's `example`, `examples`, and `description`:
|
|
785
|
+
|
|
786
|
+
- Filter: `/postgrest/i` or `field.op.value` / `status.eq.active` → `postgrest`; `status=eq:` → `colon`; else `equality`. A free-text `filter` string that still looks like equality produces an `x-query` gap telling you to declare the grammar.
|
|
787
|
+
- Sort: `name.asc` → dotted; `name:desc` → colon; `name desc` → spaced; leading `-field` → prefixed; else dotted.
|
|
788
|
+
- Select: parameter name or description contains `fields[…]` → bracketed; else csv.
|
|
789
|
+
|
|
790
|
+
Without `x-query`, if a filter/order/select/search role resolves, oat assumes **every scalar** is filterable/sortable/selectable (`string` / `number` / `integer` / `boolean`, including nullable unions). Searchable-without-tag is further narrowed to names matching `name|title|slug|label|description|email`. `doctor` warns. Pagination-only lists stay uncovered.
|
|
791
|
+
|
|
792
|
+
Searchable / filterable / sortable / selectable from the tag are used as given. `maxLimit` is also taken from a page-size parameter's `schema.maximum` when the tag omits it.
|
|
793
|
+
|
|
794
|
+
## Pagination and envelopes
|
|
795
|
+
|
|
796
|
+
Three page models, all first-class:
|
|
797
|
+
|
|
798
|
+
- **Page number** (`page` + `limit` roles).
|
|
799
|
+
- **Offset** (`offset` + `limit`). Checks that say "page 3" translate to `offset = (page - 1) * size` (size defaults to 20 only for that translation).
|
|
800
|
+
- **Cursor** (`cursor` role + envelope `nextCursor` or a `Link: rel=next` header).
|
|
801
|
+
|
|
802
|
+
`hasMore` is taken from the body (`hasMore`, `has_more`, `hasNextPage`, `more`) **or** from a documented `Link` response header. Under Link pagination, **absence** of `rel="next"` means no more pages.
|
|
803
|
+
|
|
804
|
+
Collection shape is derived from the success JSON schema, not from hardcoded wrapper names:
|
|
805
|
+
|
|
806
|
+
- Response `type: array` → the body is the list (`key: null`).
|
|
807
|
+
- Otherwise the array property whose items are objects, skipping sidecar names `error(s)`, `warning(s)`, `message(s)`, `meta`, `links`. Resource-named envelopes (`{ tables: [...] }`) work.
|
|
808
|
+
- Sibling keys become envelope fields:
|
|
809
|
+
|
|
810
|
+
| role | accepted property names |
|
|
811
|
+
| ---------- | ----------------------------------------------------------- |
|
|
812
|
+
| total | `count`, `total`, `totalCount`, `total_count`, `totalItems` |
|
|
813
|
+
| hasMore | `hasMore`, `has_more`, `hasNextPage`, `more` |
|
|
814
|
+
| nextCursor | `nextCursor`, `next_cursor`, `cursor`, `next`, `endCursor` |
|
|
815
|
+
| page | `page`, `pageNumber`, `page_number`, `offset` |
|
|
816
|
+
| limit | `limit`, `perPage`, `per_page`, `pageSize`, `page_size` |
|
|
817
|
+
|
|
818
|
+
Success schema is the first JSON media type on responses `200`, `201`, `202`, `2XX`, or `default`. Request schema is the first JSON media type on `requestBody`. Non-JSON media types are ignored.
|
|
145
819
|
|
|
146
820
|
## OpenAPI meta tags
|
|
147
821
|
|
|
148
|
-
Vendor-neutral `x-*` extensions
|
|
822
|
+
Vendor-neutral `x-*` extensions. Every one is optional. Precedence: **explicit tag → heuristic → skip with a coverage gap**.
|
|
149
823
|
|
|
150
|
-
|
|
824
|
+
A complete document with every tag in place is shipped as `labs/annotated-openapi.yaml` (also in the npm package). `oat conformance` asserts the derived model matches that file.
|
|
151
825
|
|
|
152
826
|
```bash
|
|
153
|
-
oat plan --spec openapi.yaml
|
|
154
|
-
oat doctor --spec openapi.yaml
|
|
827
|
+
oat plan --spec node_modules/@lovrozagar/oat/labs/annotated-openapi.yaml
|
|
828
|
+
oat doctor --spec node_modules/@lovrozagar/oat/labs/annotated-openapi.yaml
|
|
155
829
|
```
|
|
156
830
|
|
|
831
|
+
What each tag **unlocks** (otherwise the check cannot run):
|
|
832
|
+
|
|
833
|
+
| tag | checks unlocked |
|
|
834
|
+
| --------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
835
|
+
| `x-async` | `async.reaches-terminal-state`, `async.receipt-identifies-the-job` |
|
|
836
|
+
| `x-effects` | `effects.declared-effect-occurs` |
|
|
837
|
+
| `x-immutable` | `patch.immutable-field-rejected` |
|
|
838
|
+
| `x-invalidate` | `invalidation.declared-route-changes` (when the list names another entity) |
|
|
839
|
+
| `x-query` | `spec.declared-filterable-is-filterable`, `spec.declared-sortable-is-sortable`, `spec.declared-selectable-is-selectable` |
|
|
840
|
+
| `x-soft-delete` | `softdelete.absent-from-default-list` |
|
|
841
|
+
| `x-invite` | `auth.invite-grants-then-revokes` |
|
|
842
|
+
|
|
843
|
+
What each tag **sharpens** (the check already runs, but the verdict changes):
|
|
844
|
+
|
|
845
|
+
| tag | without it |
|
|
846
|
+
| ---------- | ---------------------------------------------------------------------------------------------- |
|
|
847
|
+
| `x-query` | every scalar is probed, including columns you never indexed — expect findings you will dismiss |
|
|
848
|
+
| `x-tenant` | a cross-tenant read is `AMBIGUITY`, not `SECURITY`, because the boundary was only inferred |
|
|
849
|
+
|
|
157
850
|
### `x-invalidate`
|
|
158
851
|
|
|
159
852
|
```yaml
|
|
@@ -162,83 +855,82 @@ x-invalidate:
|
|
|
162
855
|
- GET /v1/projects/{project_id}/tables/{table_id}
|
|
163
856
|
```
|
|
164
857
|
|
|
165
|
-
`string[]` of `"METHOD /path"`.
|
|
858
|
+
`string[]` of `"METHOD /path"`. Colon or brace path params; oat normalises to brace. Method is uppercased.
|
|
166
859
|
|
|
167
|
-
This is the entity graph. Inverted, it
|
|
860
|
+
This is the entity graph. Inverted, it is each entity's read surface. Highest-value tag.
|
|
168
861
|
|
|
169
|
-
**Fallback:** pair a mutator with
|
|
862
|
+
**Fallback:** pair a mutator with sibling collection/item routes on the same path prefix. Misses cross-entity effects.
|
|
863
|
+
|
|
864
|
+
**Unlocks:** `invalidation.declared-route-changes` (when the list names _another_ entity's route).
|
|
170
865
|
|
|
171
866
|
### `x-entity`
|
|
172
867
|
|
|
173
868
|
```yaml
|
|
174
869
|
x-entity:
|
|
175
870
|
name: table
|
|
176
|
-
action: create
|
|
177
|
-
identity: id
|
|
871
|
+
action: create # create | list | read | update | delete | action
|
|
872
|
+
identity: id
|
|
178
873
|
```
|
|
179
874
|
|
|
180
|
-
Overrides path-segment
|
|
181
|
-
|
|
182
|
-
`identity` matters when the item schema does not declare an `id` — for example free-form row objects.
|
|
875
|
+
Overrides path-segment inference. `identity` is required when the item schema has no `id` (or `uuid` / `slug` / `key` / `name`).
|
|
183
876
|
|
|
184
|
-
**Fallback:** deepest plural segment + HTTP verb.
|
|
877
|
+
**Fallback:** deepest plural segment + HTTP verb. See [How the model is derived](#how-the-model-is-derived).
|
|
185
878
|
|
|
186
879
|
### `x-invite`
|
|
187
880
|
|
|
188
881
|
```yaml
|
|
189
882
|
x-invite:
|
|
190
|
-
invite: table.invite
|
|
191
|
-
accept: invite.accept
|
|
192
|
-
revoke: table.revoke
|
|
193
|
-
granteeField: key
|
|
194
|
-
tokenPointer: $.token
|
|
195
|
-
grantPointer: $.grant_id
|
|
883
|
+
invite: table.invite
|
|
884
|
+
accept: invite.accept
|
|
885
|
+
revoke: table.revoke
|
|
886
|
+
granteeField: key
|
|
887
|
+
tokenPointer: $.token
|
|
888
|
+
grantPointer: $.grant_id
|
|
196
889
|
```
|
|
197
890
|
|
|
198
|
-
|
|
891
|
+
Put this on the invite operation. Config must give the invitee `inviteAs`. Defaults if omitted: `granteeField: key`, `tokenPointer: $.token`, `grantPointer: $.grant_id`. All three of `invite` / `accept` / `revoke` (operationIds) are required or the tag is ignored.
|
|
199
892
|
|
|
200
|
-
|
|
893
|
+
Timeline asserted: cannot read → invite → still cannot → accept → can → revoke → cannot.
|
|
201
894
|
|
|
202
|
-
**Fallback:** the check does not run.
|
|
895
|
+
**Fallback:** the check does not run.
|
|
203
896
|
|
|
204
897
|
### `x-query`
|
|
205
898
|
|
|
206
899
|
```yaml
|
|
207
900
|
x-query:
|
|
208
901
|
filterable: [id, name, slug, status, created_at]
|
|
209
|
-
sortable:
|
|
902
|
+
sortable: [name, created_at, updated_at]
|
|
210
903
|
searchable: [name, slug]
|
|
211
904
|
selectable: [id, name, slug, created_at]
|
|
212
905
|
maxLimit: 100
|
|
213
906
|
defaultOrder: created_at.desc
|
|
214
907
|
stableTiebreak: id
|
|
908
|
+
grammar: postgrest # postgrest | colon | equality
|
|
215
909
|
```
|
|
216
910
|
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
Without it, oat has to guess which fields are filterable from the item schema and will report false failures on fields the backend does not index.
|
|
911
|
+
States what `filter` / `order` / `q` / `select` actually support. `stableTiebreak` is load-bearing for keyset pagination. `grammar` pins how oat writes values (see [Query roles and grammars](#query-roles-and-grammars)).
|
|
220
912
|
|
|
221
|
-
|
|
913
|
+
**Fallback:** if those roles resolve, treat every scalar as capable and warn.
|
|
222
914
|
|
|
223
|
-
**
|
|
915
|
+
**Unlocks:** `spec.declared-filterable-is-filterable`, `spec.declared-sortable-is-sortable`, `spec.declared-selectable-is-selectable`. Sharpens every other query check (without the tag they probe columns you may not have indexed).
|
|
224
916
|
|
|
225
917
|
### `x-async`
|
|
226
918
|
|
|
227
919
|
```yaml
|
|
228
920
|
x-async:
|
|
229
921
|
poll: "GET /v1/projects/{project_id}/batches/{batch_id}"
|
|
230
|
-
idFrom: batch_id
|
|
922
|
+
idFrom: batch_id # or $.id
|
|
231
923
|
until: "status.in.complete,partial,failed"
|
|
232
924
|
successWhen: "status.eq.complete"
|
|
233
925
|
timeoutMs: 120000
|
|
234
926
|
pollIntervalMs: 2000
|
|
235
927
|
```
|
|
236
928
|
|
|
237
|
-
|
|
929
|
+
The HTTP response is a receipt. oat polls until `until` matches, then treats that payload as the result. `poll` may be an operationId or `"GET /path/{id}"`. Defaults: `timeoutMs: 120000`, `pollIntervalMs: 2000`.
|
|
238
930
|
|
|
239
|
-
|
|
931
|
+
`until` / `successWhen` use the same `field.op.value` predicates as filters (`eq`, `in`, …).
|
|
240
932
|
|
|
241
|
-
**Fallback:** treated as synchronous;
|
|
933
|
+
**Fallback:** treated as synchronous; async checks are `COVERAGE_GAP`.
|
|
242
934
|
|
|
243
935
|
### `x-effects`
|
|
244
936
|
|
|
@@ -248,13 +940,9 @@ x-effects:
|
|
|
248
940
|
- { entity: delivery, op: append, count: 1 }
|
|
249
941
|
```
|
|
250
942
|
|
|
251
|
-
`
|
|
252
|
-
|
|
253
|
-
Also tells oat to track and clean up resources created as side effects.
|
|
254
|
-
|
|
255
|
-
`op`: `create` | `append` | `update` | `delete` | `replace`.
|
|
943
|
+
`op`: `create` | `append` | `update` | `delete` | `replace`. oat asserts an exact cardinality delta on the named entity's list. `count` defaults to `1` when omitted.
|
|
256
944
|
|
|
257
|
-
**Fallback:** derived from `x-entity.action` for
|
|
945
|
+
**Fallback:** derived from `x-entity.action` for this entity only.
|
|
258
946
|
|
|
259
947
|
### `x-soft-delete`
|
|
260
948
|
|
|
@@ -262,9 +950,7 @@ Also tells oat to track and clean up resources created as side effects.
|
|
|
262
950
|
x-soft-delete: deleted_at
|
|
263
951
|
```
|
|
264
952
|
|
|
265
|
-
On
|
|
266
|
-
|
|
267
|
-
Changes the post-delete assertion from "absent everywhere" to "absent from the default list". Without this tag, correct soft-delete behaviour is reported as a bug.
|
|
953
|
+
On any operation of the entity (commonly DELETE). Tombstone, not remove. Without it, a correct soft-delete looks like a bug (the row is still GET-able).
|
|
268
954
|
|
|
269
955
|
### `x-immutable` / `x-generated`
|
|
270
956
|
|
|
@@ -273,9 +959,9 @@ x-immutable: [id, project_id, created_at]
|
|
|
273
959
|
x-generated: [id, created_at, updated_at]
|
|
274
960
|
```
|
|
275
961
|
|
|
276
|
-
|
|
962
|
+
Generated fields are omitted from create bodies and expected in responses. Immutable fields must reject or ignore PATCH.
|
|
277
963
|
|
|
278
|
-
**Fallback:** `readOnly: true`
|
|
964
|
+
**Fallback:** `readOnly: true` counts as generated. No immutability testing without the tag.
|
|
279
965
|
|
|
280
966
|
### `x-tenant`
|
|
281
967
|
|
|
@@ -283,118 +969,524 @@ x-generated: [id, created_at, updated_at]
|
|
|
283
969
|
x-tenant: project_id
|
|
284
970
|
```
|
|
285
971
|
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
Drives the isolation matrix: a second principal in another tenant must not see, read, or mutate the record — and `filter=id.eq.<other tenant's id>` must return empty rather than becoming an authz bypass.
|
|
972
|
+
Path parameter that scopes the operation.
|
|
289
973
|
|
|
290
|
-
**Fallback:** regex over `{organization_id}`, `{project_id}`, `{tenant_id}`, `{workspace_id}`, `{app_slug}
|
|
974
|
+
**Fallback:** regex over `{organization_id}`, `{project_id}`, `{tenant_id}`, `{workspace_id}`, `{app_slug}`, `{org_id}`, `{account_id}`, … (`org|organization|tenant|workspace|account|project|app` + optional `_id`/`_slug`). Without the tag, a cross-tenant read is `AMBIGUITY`, not `SECURITY`.
|
|
291
975
|
|
|
292
976
|
### `x-root`
|
|
293
977
|
|
|
294
978
|
```yaml
|
|
979
|
+
# on a path parameter, not an operation
|
|
295
980
|
x-root: true
|
|
296
981
|
```
|
|
297
982
|
|
|
298
|
-
|
|
983
|
+
This resource has no create endpoint; supply it in config `roots` / principal `roots`.
|
|
299
984
|
|
|
300
|
-
**Fallback:** inferred when a path param has no
|
|
985
|
+
**Fallback:** inferred when a path param has no create op; everything beneath is `UNSEEDABLE`.
|
|
301
986
|
|
|
302
|
-
### `x-
|
|
987
|
+
### `x-cleanup`
|
|
303
988
|
|
|
304
989
|
```yaml
|
|
305
|
-
x-
|
|
306
|
-
x-destructive: true # excluded outside --profile paranoid
|
|
307
|
-
x-idempotent: true # oat asserts repeat-equivalence
|
|
990
|
+
x-cleanup: "DELETE /v1/projects/{project_id}/tables/{table_id}"
|
|
308
991
|
```
|
|
309
992
|
|
|
310
|
-
|
|
993
|
+
Teardown route when the entity has no discoverable delete. Without it, leftover records are reported at end of run.
|
|
311
994
|
|
|
312
|
-
### `x-
|
|
995
|
+
### `x-cost` / `x-destructive` / `x-idempotent` / `x-fresh-principal`
|
|
313
996
|
|
|
314
997
|
```yaml
|
|
315
|
-
x-
|
|
998
|
+
x-cost: high # low | medium | high
|
|
999
|
+
x-destructive: true
|
|
1000
|
+
x-idempotent: true
|
|
1001
|
+
x-fresh-principal: true
|
|
316
1002
|
```
|
|
317
1003
|
|
|
318
|
-
|
|
1004
|
+
These are **parsed onto the operation model**. They are **not yet consulted by `oat run`** — there is no `--profile`. Document them for the model/`plan` output. Replay safety is tested from a documented `Idempotency-Key` header (`idempotency.replay-does-not-duplicate`), not from `x-idempotent`.
|
|
1005
|
+
|
|
1006
|
+
## Checks
|
|
1007
|
+
|
|
1008
|
+
55 checks. A check that cannot run says so (`did not apply` + `needs`). A check that depends on a broken primitive is `BLOCKED`. A check that ran and stopped is inconclusive, not a pass.
|
|
1009
|
+
|
|
1010
|
+
Order is fixed (foundations first) so cascade suppression has a cause to point at. Mutating checks run alone; read-only checks may share in-flight requests under `maxInFlight`.
|
|
1011
|
+
|
|
1012
|
+
`depends` is the `dependsOn` list: if any of those already failed **for this entity**, this check is `BLOCKED` rather than reported as a second defect. Suppression is transitive.
|
|
1013
|
+
|
|
1014
|
+
| id | asserts | needs | depends |
|
|
1015
|
+
| ------------------------------------------ | ------------------------------------------------------------------------------------ | --------------------------------------------------------- | --------------------------------------- |
|
|
1016
|
+
| `list.read-after-write` | a just-created record appears on the list | create + a seeded record | — |
|
|
1017
|
+
| `create.persists-submitted-fields` | every writable field sent on create is echoed | create that echoes the record | `list.read-after-write` |
|
|
1018
|
+
| `create.status-matches-document` | create status is one the document declared | create | — |
|
|
1019
|
+
| `schema.success-response-matches-document` | create body validates against the success schema | success schema on create | `create.status-matches-document` |
|
|
1020
|
+
| `schema.error-response-matches-document` | an error body validates against the documented error schema | error schema on the item route | — |
|
|
1021
|
+
| `pagination.limit-bounds-page-size` | page size ≤ the requested limit | page-size _role_ (aliases include `limit`, `per_page`, …) | `list.read-after-write` |
|
|
1022
|
+
| `pagination.limit-respects-documented-max` | requesting more than `maxLimit` does not return more | declared `maxLimit` and a larger cohort | `pagination.limit-bounds-page-size` |
|
|
1023
|
+
| `pagination.has-more-is-accurate` | `hasMore` / `Link rel=next` matches whether another page exists | page-forward + `hasMore` or `Link rel=next` | `pagination.limit-bounds-page-size` |
|
|
1024
|
+
| `pagination.page-walk-covers-set` | walking pages covers the collection with no gaps or dupes | page or offset + ≥3 records | `pagination.limit-bounds-page-size` |
|
|
1025
|
+
| `pagination.cursor-agrees-with-page` | cursor walk and page walk yield the same set | both cursor and page | `pagination.limit-bounds-page-size` |
|
|
1026
|
+
| `filter.unknown-field-rejected` | a filter on a field that does not exist is not silently ignored | a filter expression | — |
|
|
1027
|
+
| `filter.equality-selects-exactly-one` | `id.eq.<one>` returns that one record | equality on the identity | `list.read-after-write` |
|
|
1028
|
+
| `filter.zero-match-returns-none` | a filter that matches nothing returns an empty page, not the whole set | same | `list.read-after-write` |
|
|
1029
|
+
| `filter.negation-partitions-the-set` | `eq` ∪ `neq` = whole set, intersection empty | eq and neq | `list.read-after-write`, equality |
|
|
1030
|
+
| `filter.and-composes-as-intersection` | `and(A,B)` = A ∩ B | two filterable fields + AND | equality / list |
|
|
1031
|
+
| `filter.or-composes-as-union` | `or(A,B)` = A ∪ B | `or()` — **postgrest grammar only** | equality / list |
|
|
1032
|
+
| `filter.like-metacharacters-escaped` | `%` `_` `*` in a value are literals, not wildcards | like operator | `list.read-after-write` |
|
|
1033
|
+
| `filter.numeric-comparison-is-numeric` | `gt`/`lt` on a number uses numeric order, not TEXT (`1,10,2`) | numeric field + a filter param | `list.read-after-write`, unknown-field |
|
|
1034
|
+
| `error.malformed-filter-not-5xx` | garbage filter text is 4xx, never 5xx | a filter expression | — |
|
|
1035
|
+
| `query.filter-selects-from-whole-set` | a filter is applied to the collection, not to the current page | filterable + ≥3 records | list / walk |
|
|
1036
|
+
| `sort.order-is-applied` | requesting a sort actually rearranges the page | order + a sortable field | `pagination.limit-bounds-page-size` |
|
|
1037
|
+
| `sort.reverse-symmetry` | desc is the reverse of asc (nulls included) | order + asc/desc | order-is-applied |
|
|
1038
|
+
| `search.q-narrows-result` | a search term that matches one record does not return the whole set | search param + searchable fields | `list.read-after-write` |
|
|
1039
|
+
| `select.projection-honoured` | `select=id,name` does not return undeclared fields | select param | — |
|
|
1040
|
+
| `count.consistent-with-returned-page` | envelope total ≥ rows on this page, and is not zero when the page is not | envelope total | `list.read-after-write` |
|
|
1041
|
+
| `count.matches-filtered-set` | filtered total equals the size of the filtered walk | total + a filter | list, equality |
|
|
1042
|
+
| `query.axes-compose` | filter + sort together: filter still holds on the sorted page | filterable + sortable | filter + sort foundations |
|
|
1043
|
+
| `query.filter-and-select-compose` | filter + select together | filterable + select | same |
|
|
1044
|
+
| `query.search-and-filter-compose` | search + filter together | filterable + search | same |
|
|
1045
|
+
| `query.filter-sort-select-compose` | filter + sort + select | filter + sort + select | same |
|
|
1046
|
+
| `query.filter-search-sort-compose` | filter + search + sort | filter + search + sort | same |
|
|
1047
|
+
| `query.filter-search-select-compose` | filter + search + select | filter + search + select | same |
|
|
1048
|
+
| `spec.declared-filterable-is-filterable` | every `x-query.filterable` field actually accepts a filter | `x-query` naming filterable fields | filter foundations |
|
|
1049
|
+
| `spec.declared-sortable-is-sortable` | every `x-query.sortable` field actually accepts a sort | `x-query` naming sortable fields | sort foundations |
|
|
1050
|
+
| `spec.declared-selectable-is-selectable` | every `x-query.selectable` field actually accepts a select | `x-query` naming selectable fields | select |
|
|
1051
|
+
| `tenant.item-not-readable-cross-tenant` | principal B cannot GET principal A's item | second principal, different `roots` | — |
|
|
1052
|
+
| `tenant.denial-does-not-reveal-existence` | 404 vs 403 (or equivalent) does not distinguish "exists other tenant" from "missing" | second principal | `tenant.item-not-readable-cross-tenant` |
|
|
1053
|
+
| `tenant.filter-does-not-bypass-scope` | `filter=id.eq.<other tenant>` does not return that row | second principal + a filter | `query.filter-selects-from-whole-set` |
|
|
1054
|
+
| `auth.rank-is-monotonic` | a lower rank cannot do what a higher rank is denied | two same-tenant principals at different `rank` | `list.read-after-write` |
|
|
1055
|
+
| `auth.invite-grants-then-revokes` | invite → accept grants; revoke takes it back | `x-invite` + peer with `inviteAs` | list, cross-tenant |
|
|
1056
|
+
| `patch.immutable-field-rejected` | PATCHing an `x-immutable` field is rejected or ignored | `x-immutable` | — |
|
|
1057
|
+
| `softdelete.absent-from-default-list` | a soft-deleted row is gone from the default list | `x-soft-delete` | `list.read-after-write` |
|
|
1058
|
+
| `invalidation.declared-route-changes` | after a write, the other entity's listed route actually changes | `x-invalidate` naming another entity | list, persist |
|
|
1059
|
+
| `effects.declared-effect-occurs` | `x-effects` cardinality delta is observed on the named list | `x-effects` | `list.read-after-write` |
|
|
1060
|
+
| `async.reaches-terminal-state` | polling `x-async` reaches `until` before `timeoutMs` | `x-async` | — |
|
|
1061
|
+
| `async.receipt-identifies-the-job` | `idFrom` on the receipt resolves to a pollable job | `x-async` + `idFrom` | — |
|
|
1062
|
+
| `patch.minimality` | PATCH `{ name }` does not clear other writable fields | update + item route | — |
|
|
1063
|
+
| `idempotency.replay-does-not-duplicate` | same Idempotency-Key + same body does not create a second row | create + documented Idempotency-Key header | list, persist |
|
|
1064
|
+
| `delete.absent-record-returns-404` | DELETE of a missing id is 404, not 200 | delete | — |
|
|
1065
|
+
| `concurrency.no-lost-update` | two PATCHes to different fields do not clobber each other | update + two writable strings | persist + patch |
|
|
1066
|
+
| `validation.enum-enforced` | a value outside the enum is rejected | enum in the request schema | — |
|
|
1067
|
+
| `validation.max-length-enforced` | a string over `maxLength` is rejected | maxLength | — |
|
|
1068
|
+
| `validation.required-enforced` | omitting a required field is rejected | required field | — |
|
|
1069
|
+
| `validation.content-type-enforced` | a wrong Content-Type is 415 when 415 is documented | documented 415 | — |
|
|
1070
|
+
| `consistency.projections-agree` | list, item, and filtered views of the same field agree | item route + a comparable field | list + persist + filter |
|
|
1071
|
+
|
|
1072
|
+
On a typical untagged CRUD document (create, list, item, `page`/`limit`, maybe `sort`):
|
|
1073
|
+
|
|
1074
|
+
- Foundations, PATCH/delete, and schema checks usually run.
|
|
1075
|
+
- The query matrix runs when filter/order/select/search roles resolve.
|
|
1076
|
+
- Isolation runs only if you configured two principals.
|
|
1077
|
+
- Spec-as-adversary and tagged behaviour never run — `doctor` says so.
|
|
1078
|
+
|
|
1079
|
+
Worked examples of what a finding looks like:
|
|
1080
|
+
|
|
1081
|
+
```text
|
|
1082
|
+
BACKEND_BUG table created row missing from GET /tables
|
|
1083
|
+
list.read-after-write
|
|
1084
|
+
|
|
1085
|
+
BACKEND_BUG product PATCH { name } cleared description
|
|
1086
|
+
patch.minimality
|
|
1087
|
+
|
|
1088
|
+
SECURITY table GET /tables/{id} readable with the other tenant's key
|
|
1089
|
+
tenant.item-not-readable-cross-tenant
|
|
1090
|
+
|
|
1091
|
+
SPEC_BUG table x-query.filterable lists "ghost"; filter=ghost.eq.x is 400
|
|
1092
|
+
spec.declared-filterable-is-filterable
|
|
1093
|
+
|
|
1094
|
+
COVERAGE_GAP batch no x-async; receipt treated as the result
|
|
1095
|
+
async.reaches-terminal-state
|
|
1096
|
+
```
|
|
319
1097
|
|
|
320
|
-
|
|
1098
|
+
## Verdicts, skips, and exit codes
|
|
321
1099
|
|
|
322
|
-
|
|
1100
|
+
| verdict | meaning | fails `oat run`? |
|
|
1101
|
+
| -------------- | -------------------------------------------------------------------- | ---------------- |
|
|
1102
|
+
| `BACKEND_BUG` | the handler is wrong | yes |
|
|
1103
|
+
| `SPEC_BUG` | the document is wrong (or disagrees with the handler) | yes |
|
|
1104
|
+
| `SECURITY` | isolation/authz failure and `x-tenant` (or equivalent) was declared | yes |
|
|
1105
|
+
| `AMBIGUITY` | same evidence, but the tenant boundary was only inferred | yes |
|
|
1106
|
+
| `COVERAGE_GAP` | the check could not run; the report names the missing tag or surface | no |
|
|
1107
|
+
| `BLOCKED` | a check this one depends on already failed | no |
|
|
323
1108
|
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
1109
|
+
Separate from findings:
|
|
1110
|
+
|
|
1111
|
+
- **did not apply** — entity never had what `needs` lists (printed in the report, not a pass). Example: no `select` parameter → `select.projection-honoured` did not apply.
|
|
1112
|
+
- **inconclusive** — the check ran and stopped (empty listing, probe 4xx, no shared filterable field, …). Not a pass. The report prints the reason.
|
|
1113
|
+
|
|
1114
|
+
Coverage is split **never** (zero entities could run it) vs **partial** (ran on some entities, skipped on others). A clean run still prints both. "Nothing found" and "nothing was looked for" are different.
|
|
327
1115
|
|
|
328
|
-
|
|
1116
|
+
Cascade suppression is transitive: one root cause is one finding, not a page of consequences. A blocked check has **not** been verified; re-run after the cause is fixed.
|
|
329
1117
|
|
|
330
|
-
|
|
1118
|
+
`oat run` exit `1` if any finding has a failing verdict. Gaps and blocked entries do not fail CI.
|
|
331
1119
|
|
|
332
|
-
|
|
1120
|
+
Console (stdout) after a run:
|
|
1121
|
+
|
|
1122
|
+
```
|
|
1123
|
+
50 checks · 5 entities · 842 requests · 41.2s · p95 90ms · 4 checks did not apply
|
|
333
1124
|
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
| Query, single axis | filter, sort, search, select, cursor vs page, count |
|
|
338
|
-
| Query, composition | filter+sort, filter+select, search+filter, and the triples |
|
|
339
|
-
| Spec as adversary | declared filterable / sortable / selectable fields actually work |
|
|
340
|
-
| Isolation | cross-tenant item, filter, and existence disclosure |
|
|
341
|
-
| Authorization | rank is monotonic; invite grants then revokes |
|
|
342
|
-
| Tagged behaviour | immutable, soft-delete, invalidate, effects, async |
|
|
343
|
-
| Validation | enum, maxLength, required, content-type |
|
|
344
|
-
| Writes | PATCH minimality, idempotency, delete 404, lost update |
|
|
1125
|
+
BACKEND DEFECTS (1)
|
|
1126
|
+
product PATCH { name } also cleared description
|
|
1127
|
+
patch.minimality
|
|
345
1128
|
|
|
346
|
-
|
|
1129
|
+
DID NOT APPLY — no entity had what these need
|
|
1130
|
+
async.reaches-terminal-state needs an operation declaring x-async
|
|
347
1131
|
|
|
348
|
-
|
|
1132
|
+
leftover teardown printed next
|
|
1133
|
+
report / matrix / graph / progress paths
|
|
1134
|
+
```
|
|
349
1135
|
|
|
350
1136
|
## Reports
|
|
351
1137
|
|
|
352
|
-
|
|
1138
|
+
Written under `--out` (default `./oat-out`):
|
|
1139
|
+
|
|
1140
|
+
| file | |
|
|
1141
|
+
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
1142
|
+
| `oat-report.md` | human report: summary, findings with request/response excerpts, coverage, latency p50/p95/max |
|
|
1143
|
+
| `oat-report.json` | same data for CI |
|
|
1144
|
+
| `matrix.html` | visual matrix of entities × checks |
|
|
1145
|
+
| `matrix.json` | the same graph (AI-friendly), including a mermaid string |
|
|
1146
|
+
| `issue-repro/*.sh` | one executable `curl` script per finding that has evidence. Directory is omitted when the run is clean. Old `repro/` is deleted at the start of each run |
|
|
1147
|
+
| `progress.log` | logfmt, one event per line, never truncated |
|
|
1148
|
+
| `progress.jsonl` | same events as JSON |
|
|
1149
|
+
| `progress.tsv` | same columns, tab-separated |
|
|
1150
|
+
| `progress.json` | latest snapshot only (overwritten ~1s) |
|
|
1151
|
+
|
|
1152
|
+
There is no HAR file. The JSON report and issue-repro scripts carry the exchanges.
|
|
1153
|
+
|
|
1154
|
+
### `oat-report.json`
|
|
1155
|
+
|
|
1156
|
+
```json
|
|
1157
|
+
{
|
|
1158
|
+
"backend": "https://api.example.com",
|
|
1159
|
+
"generatedAt": "2026-08-15T12:00:00.000Z",
|
|
1160
|
+
"durationMs": 41200,
|
|
1161
|
+
"requests": 842,
|
|
1162
|
+
"entitiesTested": ["store", "product"],
|
|
1163
|
+
"checksRun": ["list.read-after-write", "patch.minimality"],
|
|
1164
|
+
"checksSkipped": [{ "check": "async.reaches-terminal-state", "entity": "store", "needs": "…" }],
|
|
1165
|
+
"checksSuppressed": [{ "check": "query.axes-compose", "entity": "store", "because": "list.read-after-write" }],
|
|
1166
|
+
"inconclusive": [{ "check": "filter.and-composes-as-intersection", "entity": "store", "reason": "…" }],
|
|
1167
|
+
"summary": { "BACKEND_BUG": 1 },
|
|
1168
|
+
"coverage": {
|
|
1169
|
+
"neverApplied": ["async.reaches-terminal-state"],
|
|
1170
|
+
"partial": [{ "check": "select.projection-honoured", "ran": 1, "skipped": 1 }]
|
|
1171
|
+
},
|
|
1172
|
+
"latency": {
|
|
1173
|
+
"p50": 12,
|
|
1174
|
+
"p95": 90,
|
|
1175
|
+
"max": 400,
|
|
1176
|
+
"slowest": { "method": "GET", "path": "/v1/products" }
|
|
1177
|
+
},
|
|
1178
|
+
"findings": [
|
|
1179
|
+
{
|
|
1180
|
+
"check": "patch.minimality",
|
|
1181
|
+
"verdict": "BACKEND_BUG",
|
|
1182
|
+
"entity": "product",
|
|
1183
|
+
"summary": "PATCH { name } also cleared description",
|
|
1184
|
+
"detail": "…",
|
|
1185
|
+
"evidence": [
|
|
1186
|
+
{
|
|
1187
|
+
"method": "PATCH",
|
|
1188
|
+
"url": "https://api.example.com/v1/products/p1",
|
|
1189
|
+
"status": 200,
|
|
1190
|
+
"requestBody": { "name": "x" },
|
|
1191
|
+
"responseBody": { "name": "x", "description": null }
|
|
1192
|
+
}
|
|
1193
|
+
]
|
|
1194
|
+
}
|
|
1195
|
+
]
|
|
1196
|
+
}
|
|
1197
|
+
```
|
|
1198
|
+
|
|
1199
|
+
Latency is reported, never asserted. oat has no baseline for "too slow".
|
|
1200
|
+
|
|
1201
|
+
Gate in CI on process exit code, or on `findings` whose `verdict` is not `COVERAGE_GAP` / `BLOCKED`.
|
|
353
1202
|
|
|
354
|
-
|
|
355
|
-
| --- | --- |
|
|
356
|
-
| `oat-out/oat-report.md` | human report |
|
|
357
|
-
| `oat-out/oat-report.json` | CI / machine copy |
|
|
358
|
-
| `oat-out/issue-repro/*.sh` | one `curl` script per finding — omitted when the run is clean |
|
|
359
|
-
| `oat-out/progress.log` | live logfmt progress (also `.jsonl` and `.tsv`) |
|
|
1203
|
+
### `matrix.json`
|
|
360
1204
|
|
|
361
|
-
|
|
1205
|
+
```json
|
|
1206
|
+
{
|
|
1207
|
+
"kind": "oat.matrix",
|
|
1208
|
+
"version": 2,
|
|
1209
|
+
"baseUrl": "…",
|
|
1210
|
+
"generatedAt": "…",
|
|
1211
|
+
"thesis": "…",
|
|
1212
|
+
"summary": "…",
|
|
1213
|
+
"index": { "entityCount": 5, "failed": ["product"], "parents": ["store"], "crossClaims": 1, "inbound": {} },
|
|
1214
|
+
"counts": { "failed": 1, "blocked": 0, "held": 40, "skipped": 14 },
|
|
1215
|
+
"entities": [
|
|
1216
|
+
{
|
|
1217
|
+
"name": "product",
|
|
1218
|
+
"identity": "id",
|
|
1219
|
+
"readSurface": ["GET /v1/products", "GET /v1/products/{id}"],
|
|
1220
|
+
"counts": { "failed": 1, "blocked": 0, "held": 20, "skipped": 5 },
|
|
1221
|
+
"roots": ["store_id"],
|
|
1222
|
+
"nodes": [
|
|
1223
|
+
{
|
|
1224
|
+
"id": "product/patch.minimality",
|
|
1225
|
+
"group": "product",
|
|
1226
|
+
"layer": "weft",
|
|
1227
|
+
"status": "failed",
|
|
1228
|
+
"verdict": "BACKEND_BUG",
|
|
1229
|
+
"summary": "…"
|
|
1230
|
+
}
|
|
1231
|
+
]
|
|
1232
|
+
}
|
|
1233
|
+
],
|
|
1234
|
+
"invalidate": [
|
|
1235
|
+
{
|
|
1236
|
+
"fromEntity": "product",
|
|
1237
|
+
"fromOp": "product.create",
|
|
1238
|
+
"toEntity": "store",
|
|
1239
|
+
"toRoute": "GET /v1/stores/{id}",
|
|
1240
|
+
"cross": true
|
|
1241
|
+
}
|
|
1242
|
+
],
|
|
1243
|
+
"edges": [{ "from": "product/list.read-after-write", "to": "product/patch.minimality", "kind": "dependsOn" }],
|
|
1244
|
+
"mermaid": "flowchart LR\n…"
|
|
1245
|
+
}
|
|
1246
|
+
```
|
|
362
1247
|
|
|
363
|
-
|
|
1248
|
+
Cell `status`: `held` (passed), `failed`, `blocked`, `skipped`. Edge `kind`: `dependsOn` (cascade) or `uses` (a later check that builds on a warp primitive).
|
|
364
1249
|
|
|
365
|
-
|
|
1250
|
+
### `issue-repro/<entity>-<check>.sh`
|
|
366
1251
|
|
|
367
|
-
|
|
1252
|
+
Created only when a finding has HTTP evidence. Not created on a clean run.
|
|
368
1253
|
|
|
369
1254
|
```bash
|
|
370
|
-
|
|
371
|
-
oat
|
|
372
|
-
|
|
1255
|
+
#!/usr/bin/env bash
|
|
1256
|
+
# Generated by oat. Set TOKEN to a valid credential before running.
|
|
1257
|
+
# product — PATCH { name } also cleared description
|
|
1258
|
+
set -u
|
|
1259
|
+
BASE="${BASE:-https://api.example.com}"
|
|
1260
|
+
TOKEN="${TOKEN:?set TOKEN to a valid credential}"
|
|
1261
|
+
|
|
1262
|
+
# step 1 — observed 200
|
|
1263
|
+
curl -sS -X PATCH "$BASE/v1/products/p1" \
|
|
1264
|
+
-H "authorization: Bearer $TOKEN" \
|
|
1265
|
+
-H "content-type: application/json" \
|
|
1266
|
+
-d '{"name":"x"}'
|
|
373
1267
|
```
|
|
374
1268
|
|
|
375
|
-
|
|
1269
|
+
`authorization`, `cookie`, and `x-api-key` are redacted to `$TOKEN`. Replay needs a live credential; ids in the script are whatever the failing run created (gone if teardown ran). Use `--keep-fixtures` when you want to replay against the same rows.
|
|
1270
|
+
|
|
1271
|
+
## Progress logs
|
|
1272
|
+
|
|
1273
|
+
`progress.log` starts with a glossary. Fields:
|
|
376
1274
|
|
|
377
|
-
|
|
1275
|
+
| key | |
|
|
1276
|
+
| -------------------------- | ------------------------------------------------------------ |
|
|
1277
|
+
| `ts` | ISO-8601 UTC when the line was written |
|
|
1278
|
+
| `status` | `ok` or `stall` (`idle_ms` ≥ 15000) |
|
|
1279
|
+
| `done` / `total` | entity index / how many entities |
|
|
1280
|
+
| `phase` | `load` \| `auth` \| `seed` \| `test` \| `teardown` \| `done` |
|
|
1281
|
+
| `entity` / `check` | current entity and check id |
|
|
1282
|
+
| `msg` | phase note |
|
|
1283
|
+
| `req` / `find` | request count, finding count |
|
|
1284
|
+
| `method` / `path` / `http` | last HTTP call |
|
|
1285
|
+
| `last_ms` | duration of that call |
|
|
1286
|
+
| `idle_ms` | ms since it returned — if this climbs, the run is stuck |
|
|
1287
|
+
| `elapsed_ms` | wall clock since start |
|
|
378
1288
|
|
|
379
|
-
|
|
1289
|
+
`--quiet` keeps the files and drops stderr.
|
|
380
1290
|
|
|
381
|
-
|
|
1291
|
+
Lines are written when the phase/entity/check/message changes, or every 2 s, or on `load`/`done`. `progress.json` is rewritten about once a second.
|
|
382
1292
|
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
1293
|
+
If `idle_ms` climbs through a long poll (`x-async`) that is expected. If it climbs on a simple GET, the process or the network is stuck. oat's own `fetch` has **no timeout**.
|
|
1294
|
+
|
|
1295
|
+
## Programmatic API
|
|
1296
|
+
|
|
1297
|
+
```ts
|
|
1298
|
+
import {
|
|
1299
|
+
defineConfig,
|
|
1300
|
+
loadConfig,
|
|
1301
|
+
run,
|
|
1302
|
+
loadSpec,
|
|
1303
|
+
dereference,
|
|
1304
|
+
buildModel,
|
|
1305
|
+
renderJson,
|
|
1306
|
+
renderMarkdown,
|
|
1307
|
+
} from "@lovrozagar/oat"
|
|
1308
|
+
|
|
1309
|
+
const config = defineConfig({
|
|
1310
|
+
spec: "./openapi.yaml",
|
|
1311
|
+
baseUrl: "https://api.example.com",
|
|
1312
|
+
principals: [{ id: "alpha", headers: { authorization: `Bearer ${process.env.API_TOKEN}` } }],
|
|
1313
|
+
})
|
|
1314
|
+
|
|
1315
|
+
const result = await run({
|
|
1316
|
+
spec: config.spec,
|
|
1317
|
+
baseUrl: config.baseUrl,
|
|
1318
|
+
principals: config.principals,
|
|
1319
|
+
seed: 1,
|
|
1320
|
+
concurrency: 1,
|
|
1321
|
+
maxInFlight: 4,
|
|
1322
|
+
onProgress: (snap) => {
|
|
1323
|
+
// snap.phase, snap.entity, snap.check, snap.message, …
|
|
1324
|
+
},
|
|
1325
|
+
})
|
|
1326
|
+
|
|
1327
|
+
// result.findings, result.checksSkipped, result.checksSuppressed,
|
|
1328
|
+
// result.inconclusive, result.entitiesTested, result.teardown, result.created
|
|
1329
|
+
```
|
|
1330
|
+
|
|
1331
|
+
`loadConfig(path)` loads `.ts` / `.js` / `.mjs` / `.json` the same way the CLI does. The CLI then expands `${NAME}` in every string. `defineConfig` is an identity function for typing; it does not interpolate. If you call `run()` with an in-process object, resolve secrets yourself (template literals, `process.env`) before passing it.
|
|
1332
|
+
|
|
1333
|
+
`run(options)` does not write files. The CLI writes reports after `run` returns. To produce the same artifacts, call `renderMarkdown` / `renderJson` / `renderMatrixHtml` / `renderMatrixGraph` / `renderRepros` with a `ReportInput` (`findings`, `model`, `client`, `baseUrl`, `entitiesTested`, `checksRun`, `startedAt`, `durationMs`, plus optional skip/suppress/inconclusive lists).
|
|
1334
|
+
|
|
1335
|
+
Offline:
|
|
1336
|
+
|
|
1337
|
+
```ts
|
|
1338
|
+
const doc = await loadSpec("./openapi.yaml")
|
|
1339
|
+
const { doc: resolved, externalRefs } = dereference(doc)
|
|
1340
|
+
const model = buildModel(resolved)
|
|
1341
|
+
```
|
|
1342
|
+
|
|
1343
|
+
Types exported: `OatConfig`, `Principal`, `AuthFlow`, `AuthStep`, `Hooks`, `RunOptions`, `RunResult`, `Finding`, `Verdict`, `Actor`, `SpecModel`, `EntityModel`, `OperationModel`, `OpenApiDocument`, matrix types.
|
|
1344
|
+
|
|
1345
|
+
## CI
|
|
1346
|
+
|
|
1347
|
+
This repository runs [`.github/workflows/ci.yml`](./.github/workflows/ci.yml) on every push and pull request to `main`: format, lint, typecheck, then `npm test` (conformance on memory + sqlite, plus the built-in combination smoke). D1, live labs Workers, and postgres are not in that job — D1 needs Cloudflare credentials, and the postgres reference backend is not yet a green CI gate.
|
|
1348
|
+
|
|
1349
|
+
The package is [`@lovrozagar/oat` on npm](https://www.npmjs.com/package/@lovrozagar/oat). That URL is the repository website. GitHub Releases match npm versions. Pushing a tag `vX.Y.Z` (same as `package.json` `version`) runs [`.github/workflows/release.yml`](./.github/workflows/release.yml): test, `npm publish` via trusted publishing, then a GitHub Release. Configure the trusted publisher once on npm (package Settings → Trusted Publisher → GitHub Actions, workflow `release.yml`, no environment). Do not publish to GitHub Packages — people install from the public npm registry.
|
|
1350
|
+
|
|
1351
|
+
Against **your** API:
|
|
1352
|
+
|
|
1353
|
+
```yaml
|
|
1354
|
+
# GitHub Actions sketch
|
|
1355
|
+
- run: npm i -D @lovrozagar/oat
|
|
1356
|
+
- run: oat doctor --spec "$SPEC_URL"
|
|
1357
|
+
- run: oat run --config oat.config.ts --concurrency 1
|
|
1358
|
+
env:
|
|
1359
|
+
API_TOKEN: ${{ secrets.API_TOKEN }}
|
|
1360
|
+
API_TOKEN_B: ${{ secrets.API_TOKEN_B }}
|
|
1361
|
+
```
|
|
1362
|
+
|
|
1363
|
+
Gate on exit code 1 (defects). Read `oat-out/oat-report.json` if you need to classify verdicts.
|
|
1364
|
+
|
|
1365
|
+
Nested APIs: keep `--concurrency 1`. Higher values create children while a parent page-walk is in flight and produce false `pagination.page-walk-covers-set` findings.
|
|
1366
|
+
|
|
1367
|
+
Wipe leftover rows on a shared database between runs if a previous `--keep-fixtures` or a crashed teardown left data. Leftover rows make numeric/filter checks look like type bugs (`1, 10, 2` from old TEXT-sorted leftovers mixed with a fresh numeric cohort).
|
|
1368
|
+
|
|
1369
|
+
## Reference defects (`oat serve --defects`)
|
|
1370
|
+
|
|
1371
|
+
Comma-separated. Each is one named lie the demo API can tell. Primary check is what conformance asserts; extras in parentheses are accepted additional symptoms of the same lie.
|
|
1372
|
+
|
|
1373
|
+
| defect | primary check | the lie |
|
|
1374
|
+
| ------------------------------------------- | ------------------------------------------ | ------------------------------------------------------ |
|
|
1375
|
+
| `STALE_LIST` | `list.read-after-write` | create succeeds, list does not show the row |
|
|
1376
|
+
| `CREATE_DROPS_FIELD` | `create.persists-submitted-fields` | a submitted field is dropped |
|
|
1377
|
+
| `CREATED_201_AS_200` | `create.status-matches-document` | create returns 200 when the spec says 201 |
|
|
1378
|
+
| `RESPONSE_SCHEMA_DRIFT` | `schema.success-response-matches-document` | success body does not match the schema |
|
|
1379
|
+
| `ERROR_SCHEMA_DRIFT` | `schema.error-response-matches-document` | error body does not match the schema |
|
|
1380
|
+
| `LIMIT_IGNORED` | `pagination.limit-bounds-page-size` | `limit` is accepted and ignored |
|
|
1381
|
+
| `LIMIT_EXCEEDS_MAX` | `pagination.limit-respects-documented-max` | documented max is not capped |
|
|
1382
|
+
| `HASMORE_ALWAYS_FALSE` | `pagination.has-more-is-accurate` | `hasMore` is always false |
|
|
1383
|
+
| `OFF_BY_ONE_PAGE` | `pagination.page-walk-covers-set` | page walk skips or repeats |
|
|
1384
|
+
| `UNSTABLE_SORT` | `pagination.page-walk-covers-set` | default order is not a total order |
|
|
1385
|
+
| `CURSOR_DRIFT` | `pagination.cursor-agrees-with-page` | cursor and page disagree |
|
|
1386
|
+
| `FILTER_IGNORED` | `filter.unknown-field-rejected` | unknown filter field is ignored |
|
|
1387
|
+
| `FILTER_EQ_NOT_APPLIED` | `filter.equality-selects-exactly-one` | equality filter is ignored |
|
|
1388
|
+
| `EMPTY_RESULT_RETURNS_ALL` | `filter.zero-match-returns-none` | empty match returns the whole set |
|
|
1389
|
+
| `NEQ_DROPS_NULLS` | `filter.negation-partitions-the-set` | `neq` drops nulls so the partition leaks |
|
|
1390
|
+
| `FILTER_GROUP_COMBINATOR_SWAPPED` | `filter.and-composes-as-intersection` | `and`/`or` are swapped |
|
|
1391
|
+
| `LIKE_UNESCAPED` | `filter.like-metacharacters-escaped` | `%`/`_` are wildcards in values |
|
|
1392
|
+
| `NUMERIC_COMPARED_AS_TEXT` | `filter.numeric-comparison-is-numeric` | numbers compared as strings |
|
|
1393
|
+
| `ERROR_500_ON_BAD_FILTER` | `error.malformed-filter-not-5xx` | bad filter is 500 |
|
|
1394
|
+
| `FILTER_AFTER_PAGINATION` | `query.filter-selects-from-whole-set` | filter applied after the page is cut |
|
|
1395
|
+
| `ORDER_IGNORED` | `sort.order-is-applied` | sort param is ignored |
|
|
1396
|
+
| `SORT_DESC_DROPS_NULLS` | `sort.reverse-symmetry` | desc drops nulls |
|
|
1397
|
+
| `SEARCH_IGNORED` | `search.q-narrows-result` | search param is ignored |
|
|
1398
|
+
| `SELECT_IGNORED` | `select.projection-honoured` | select param is ignored |
|
|
1399
|
+
| `COUNT_ALWAYS_ZERO` | `count.consistent-with-returned-page` | total is always 0 |
|
|
1400
|
+
| `COUNT_IGNORES_FILTER` | `count.matches-filtered-set` | total ignores the filter |
|
|
1401
|
+
| `FILTER_DROPPED_WHEN_SORTED` | `query.axes-compose` | sort drops the filter |
|
|
1402
|
+
| `FILTER_DROPPED_WHEN_SELECTED` | `query.filter-and-select-compose` | select drops the filter |
|
|
1403
|
+
| `FILTER_DROPPED_WHEN_SEARCHED` | `query.search-and-filter-compose` | search drops the filter |
|
|
1404
|
+
| `FILTER_DROPPED_WHEN_SORTED_AND_SELECTED` | `query.filter-sort-select-compose` | the triple drops the filter |
|
|
1405
|
+
| `FILTER_DROPPED_WHEN_SORTED_AND_SEARCHED` | `query.filter-search-sort-compose` | the triple drops the filter |
|
|
1406
|
+
| `FILTER_DROPPED_WHEN_SEARCHED_AND_SELECTED` | `query.filter-search-select-compose` | the triple drops the filter |
|
|
1407
|
+
| `SPEC_OVERCLAIMS_FILTERABLE` | `spec.declared-filterable-is-filterable` | `x-query` lists a field that 400s |
|
|
1408
|
+
| `SPEC_OVERCLAIMS_SORTABLE` | `spec.declared-sortable-is-sortable` | same for sort |
|
|
1409
|
+
| `SPEC_OVERCLAIMS_SELECTABLE` | `spec.declared-selectable-is-selectable` | same for select |
|
|
1410
|
+
| `CROSS_TENANT_READ` | `tenant.item-not-readable-cross-tenant` | item GET is global by id |
|
|
1411
|
+
| `EXISTENCE_LEAK_VIA_STATUS` | `tenant.denial-does-not-reveal-existence` | 403 vs 404 reveals the other tenant's row |
|
|
1412
|
+
| `TENANT_LEAK_VIA_FILTER` | `tenant.filter-does-not-bypass-scope` | filter drops the tenant predicate |
|
|
1413
|
+
| `ROLE_MONOTONICITY_BROKEN` | `auth.rank-is-monotonic` | a lower rank can do more |
|
|
1414
|
+
| `INVITE_NEVER_GRANTS` | `auth.invite-grants-then-revokes` | accept does not grant |
|
|
1415
|
+
| `REVOKE_IGNORED` | `auth.invite-grants-then-revokes` | revoke leaves the grant |
|
|
1416
|
+
| `IMMUTABLE_WRITABLE` | `patch.immutable-field-rejected` | immutable fields accept writes |
|
|
1417
|
+
| `SOFT_DELETE_LEAK` | `softdelete.absent-from-default-list` | tombstone stays on the default list |
|
|
1418
|
+
| `PARENT_PROJECTION_STALE` | `invalidation.declared-route-changes` | child write does not bump the parent |
|
|
1419
|
+
| `EFFECT_NOT_APPLIED` | `effects.declared-effect-occurs` | declared cardinality delta does not happen |
|
|
1420
|
+
| `ASYNC_NEVER_COMPLETES` | `async.reaches-terminal-state` | job stays pending |
|
|
1421
|
+
| `ASYNC_RECEIPT_MISSING_ID` | `async.receipt-identifies-the-job` | receipt has no id |
|
|
1422
|
+
| `PATCH_REPLACES` | `patch.minimality` | PATCH is implemented as replace |
|
|
1423
|
+
| `IDEMPOTENCY_IGNORED` | `idempotency.replay-does-not-duplicate` | Idempotency-Key is ignored |
|
|
1424
|
+
| `DELETE_MISSING_OK` | `delete.absent-record-returns-404` | DELETE missing returns 200 |
|
|
1425
|
+
| `CONCURRENT_WRITE_LOST` | `concurrency.no-lost-update` | full-row write clobbers a parallel PATCH |
|
|
1426
|
+
| `ENUM_NOT_VALIDATED` | `validation.enum-enforced` | enum is not enforced |
|
|
1427
|
+
| `MAXLENGTH_NOT_VALIDATED` | `validation.max-length-enforced` | maxLength is not enforced |
|
|
1428
|
+
| `REQUIRED_NOT_VALIDATED` | `validation.required-enforced` | required is not enforced |
|
|
1429
|
+
| `CONTENT_TYPE_NOT_ENFORCED` | `validation.content-type-enforced` | wrong Content-Type is accepted |
|
|
1430
|
+
| `LIST_DETAIL_DISAGREE` | `consistency.projections-agree` | list and item show different values |
|
|
1431
|
+
| `COLUMN_NAME_MISMATCH` | `create.persists-submitted-fields` | SQL identifier does not match the field (SQL backends) |
|
|
1432
|
+
| `COLLATION_INCONSISTENT` | `pagination.cursor-agrees-with-page` | cursor order ≠ page order (SQL) |
|
|
1433
|
+
|
|
1434
|
+
```bash
|
|
1435
|
+
oat serve --defects STALE_LIST,PATCH_REPLACES
|
|
1436
|
+
oat run --config labs/local.config.ts --base-url <url>
|
|
1437
|
+
```
|
|
1438
|
+
|
|
1439
|
+
`COLUMN_NAME_MISMATCH`, `NUMERIC_COMPARED_AS_TEXT`, `COLLATION_INCONSISTENT`, and `CONCURRENT_WRITE_LOST` are SQL-only in conformance (the in-memory store cannot exhibit them).
|
|
1440
|
+
|
|
1441
|
+
## Limits and non-features
|
|
1442
|
+
|
|
1443
|
+
These are deliberate. An agent should not invent a flag for them.
|
|
1444
|
+
|
|
1445
|
+
- **No request timeout.** `fetch` waits until the server answers. Watch `idle_ms`.
|
|
1446
|
+
- **No retry on 5xx / 429 / 401.** A 401 mid-run is evidence, not a refresh trigger. Refresh is expiry-based only.
|
|
1447
|
+
- **No OpenAPI `security`.** Put credentials in `principals`. Cookie auth is a `headers: { cookie: "…" }` (or a flow that sets that header).
|
|
1448
|
+
- **No `servers[]`.** Always set `baseUrl`.
|
|
1449
|
+
- **JSON request bodies only.** Multipart, form, file upload, and XML are not sent. Those operations will fail to seed or will not apply.
|
|
1450
|
+
- **No webhook / callback / link-object following.**
|
|
1451
|
+
- **External `$ref`s are not fetched.** In-document `$ref`s are.
|
|
1452
|
+
- **No `--profile` / `x-cost` gating.** High-cost operations still run if they are create/list/read/update/delete.
|
|
1453
|
+
- **`x-idempotent` is not the idempotency check.** The check keys off a documented `Idempotency-Key` header.
|
|
1454
|
+
- **Equality filter grammar** cannot express `neq` / `gt` / `like` / `and` / `or`. Those checks did-not-apply, they do not fail.
|
|
1455
|
+
- **`or()` is postgrest-only.**
|
|
1456
|
+
- **Concurrency > 1 on nested graphs invents pagination bugs.** Use `1`.
|
|
1457
|
+
- **Leftover rows on a shared DB** poison numeric and filter checks. Wipe between runs.
|
|
1458
|
+
- **`--only` uses plan names** (`store`, not `stores` or `/v1/stores`).
|
|
1459
|
+
- **Default cohort is 7.** `pagination.limit-respects-documented-max` needs `cohortSize > maxLimit`.
|
|
1460
|
+
- **First principal is the writer.** Extra principals are peers / lattice, not a pool of writers.
|
|
1461
|
+
|
|
1462
|
+
## Compared to schema fuzzers
|
|
1463
|
+
|
|
1464
|
+
Tools like [Schemathesis](https://schemathesis.readthedocs.io/) generate request bodies from the OpenAPI schema and check that each response validates, is not a 5xx, and matches a documented status.
|
|
1465
|
+
|
|
1466
|
+
oat does that on the traffic it sends: create/update bodies come from the schema; `validation.*` and `schema.*` catch drift; `error.malformed-filter-not-5xx` fails a 500. You do not need a second tool for “send OpenAPI-shaped requests and watch for 500s.”
|
|
1467
|
+
|
|
1468
|
+
What they cannot do — and what oat is for — is **state**. A fuzzer’s requests are independent. It has no model of the row it just created, so it cannot ask whether that row appears on the list, whether a filter and its negation partition the set, whether `GET` item and `GET` list agree, or whether another tenant can read it. oat keeps a shadow of everything it wrote and matrix-tests those projections against each other.
|
|
1469
|
+
|
|
1470
|
+
| | schema fuzzer | oat |
|
|
1471
|
+
| ----------------------------- | ----------------------- | ----------------------------------------------------------------- |
|
|
1472
|
+
| generate bodies from schema | yes | yes |
|
|
1473
|
+
| catch 500 / schema drift | yes | yes (`schema.*`, `validation.*`, malformed → 5xx) |
|
|
1474
|
+
| remember what it created | no | shadow model of the cohort |
|
|
1475
|
+
| same fact, every projection | no | the matrix (list / item / filter / sort / page / tenant / parent) |
|
|
1476
|
+
| filter ∩ negation = universe | no | `filter.negation-partitions-the-set` |
|
|
1477
|
+
| page walk covers the set | no | `pagination.page-walk-covers-set` |
|
|
1478
|
+
| N live principals | header injection | peer tenants + rank lattice + invite timeline |
|
|
1479
|
+
| multi-step / out-of-band auth | usually a static header | declarative chain, `resolveOutOfBand`, JWT refresh |
|
|
1480
|
+
|
|
1481
|
+
They are not a peer you should also run for coverage oat misses. The implication is one way.
|
|
1482
|
+
|
|
1483
|
+
Runtime dependencies of a run against _your_ API: `ajv`, `ajv-formats`, `yaml`. SQL drivers are optional and only loaded for `oat serve` / `oat conformance`.
|
|
394
1484
|
|
|
395
1485
|
## Labs
|
|
396
1486
|
|
|
397
|
-
[`labs/`](./labs) is a family of real Hono
|
|
1487
|
+
[`labs/`](./labs) is a family of real Hono + Cloudflare D1 backends this repo uses to iterate oat (correct worlds and planted bugs). Schema is generated from [`labs/worlds/catalog.ts`](./labs/worlds/catalog.ts). See [`labs/README.md`](./labs/README.md). You do not need labs to test your own API.
|
|
1488
|
+
|
|
1489
|
+
Shipped in the npm package for copy-paste: `labs/local.config.ts`, `labs/minimal.config.ts`, `labs/anyrow.config.ts`, `labs/annotated-openapi.yaml`.
|
|
398
1490
|
|
|
399
1491
|
## License
|
|
400
1492
|
|