@schwabyio/ata 0.0.0-stage → 0.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/SPEC.md ADDED
@@ -0,0 +1,2146 @@
1
+ # The Accelerator file format
2
+
3
+ Version 0.20.0, the version of Accelerator and `ata` that reads it: the two are released
4
+ together.
5
+
6
+ This document specifies the YAML files that **Accelerator**, the desktop app, and **`ata`**,
7
+ the command-line runner, read and write. Together they make up Accelerator Test
8
+ Automation, a tool for testing HTTP APIs. A folder that follows this document works in
9
+ both of them.
10
+
11
+ It is written for people and for coding agents alike. Every key has a table saying its
12
+ type, whether it is required and its default. Every rule that makes a file invalid is
13
+ stated where it applies, and all of them are collected in
14
+ [Appendix A](#appendix-a-validation-rules). [Appendix B](#appendix-b-a-complete-project)
15
+ is a complete, valid project to start from.
16
+
17
+ **Conventions**
18
+
19
+ - **must**, **must not**, **should** and **may** are used as in RFC 2119.
20
+ - Every file is UTF-8 YAML 1.2, one document holding a mapping, with the extension
21
+ **`.yml`**. A `.yaml` file is not read.
22
+ - Paths written in files are relative, and use `/`.
23
+ - Section numbers are stable. Accelerator's error messages cite them, as in
24
+ "(SPEC.md §2.5)".
25
+
26
+ ---
27
+
28
+ ## At a glance
29
+
30
+ | File | Where | What it is | § |
31
+ | ----------------------------- | --------------------------------------- | ------------------------------------------- | ---- |
32
+ | `collections/<id>.yml` | `collections/`, or one folder inside it | A collection: requests run in order | §2 |
33
+ | `collections/<id>.csv\|.json` | Beside its collection | A data file: the collection runs once a row | §2.8 |
34
+ | `environments/<name>.yml` | `environments/` | Variables, secrets and flags for one target | §6 |
35
+ | `project.yml` | The project folder | Name, global project, variables, trust | §1.1 |
36
+ | `settings.yml` | The project folder | How `ata` runs the project | §1.3 |
37
+ | `rules.yml` | The project folder | How the project's files are written | §1.4 |
38
+ | `requests/<id>.yml` | `requests/`, or one folder inside it | Reusable requests, run by `use:` | §2.5 |
39
+ | `endpoints/<id>.yml` | `endpoints/`, or one folder inside it | Defaults and checks per method and path | §2.6 |
40
+ | `bases/<id>.yml` | `bases/`, or one folder inside it | A base collection, for `extends:` | §2.7 |
41
+ | `checks/<name>.js` | `checks/` | Shared check functions | §5 |
42
+ | `.env` | The project folder, never committed | Values for secrets | §6 |
43
+
44
+ The smallest project `ata` runs is three files:
45
+
46
+ ```yaml
47
+ # collections/health.yml
48
+ id: health
49
+ steps:
50
+ - name: service is up
51
+ GET: '{{baseUrl}}/health'
52
+ tests: |
53
+ ata.expectResponseStatusCodeToBe(200)
54
+ ```
55
+
56
+ ```yaml
57
+ # environments/local.yml
58
+ vars:
59
+ baseUrl: http://localhost:8080
60
+ ```
61
+
62
+ ```yaml
63
+ # settings.yml: how ata runs the project (§1.3)
64
+ environmentType: local
65
+ ```
66
+
67
+ The desktop app needs only the first two, and the environment is picked in the app.
68
+
69
+ **The mistakes that come up most:**
70
+
71
+ 1. **A value that starts with `{{` must be quoted.** YAML reads an unquoted `{` as the
72
+ start of a map, so `GET: {{baseUrl}}/x` is invalid. Write `GET: '{{baseUrl}}/x'`.
73
+ 2. **A collection's `id` must equal its file name** without `.yml` (§2).
74
+ 3. **Each step has exactly one method key**, in capitals, whose value is the URL: `GET:`,
75
+ not `get:` or `method: GET` (§2.1).
76
+ 4. **A JSON body is a string**, not a YAML map: `json: |` followed by the JSON (§2.2).
77
+ 5. **Code goes in `tests` and `before.script`** as a block (`|`). Checks are calls on `ata`
78
+ (§3, §5).
79
+ 6. **`vars` hold plain values only**. Anything computed is set in `before.script` with
80
+ `ata.set` (§4).
81
+ 7. **An unknown `{{variable}}` fails the step.** It is never sent as literal text (§4).
82
+ 8. **Step tags need `stepTags: true`** on the collection (§2.4).
83
+
84
+ ---
85
+
86
+ ## 1. Layout
87
+
88
+ A **project** is a folder holding `collections/` and, usually, `environments/`:
89
+
90
+ ```
91
+ payments/ a project
92
+ ├── project.yml optional (§1.1)
93
+ ├── settings.yml how ata runs it (§1.3)
94
+ ├── rules.yml how its files are written (§1.4)
95
+ ├── collections/
96
+ │ ├── smoke.yml a collection
97
+ │ └── checkout/ a folder grouping collections: one level only
98
+ │ ├── sessions.yml
99
+ │ ├── sessions.csv its data file (§2.8)
100
+ │ └── refunds.yml
101
+ ├── environments/
102
+ │ ├── local.yml
103
+ │ └── staging.yml
104
+ ├── requests/ reusable requests (§2.5)
105
+ ├── endpoints/ endpoint bases (§2.6)
106
+ ├── bases/ base collections (§2.7)
107
+ ├── checks/ check files (§5)
108
+ ├── files/ anything a body uploads (§2.2); any name will do
109
+ └── .env secret values; not committed (§6)
110
+ ```
111
+
112
+ **Every `.yml` file directly in `collections/`, or in a folder one level down, is a
113
+ collection.** Nothing else is. Discovery is exact: no other `.yml` in a repository is
114
+ mistaken for a collection. A file that does not parse is reported as a broken
115
+ collection, never skipped in silence.
116
+
117
+ - A folder inside a folder of `collections/` is reported as a problem and not
118
+ read. `requests/`, `endpoints/` and `bases/` are read to the same depth, and
119
+ `checks/` only at its top level.
120
+ - Names starting with `.` are ignored, as are the folders `node_modules`, `.git`,
121
+ `reports`, `test-results`, `out` and `dist`.
122
+ - Folders carry no configuration and need no file of their own. They group
123
+ collections for display and for running a group.
124
+
125
+ A project is any folder: the root of a repository, or one service of a monorepo. A
126
+ monorepo is several projects, one per service, and they can share a **global project**
127
+ (§1.1):
128
+
129
+ ```
130
+ platform/ a repository, not itself a project
131
+ ├── services/auth/ a project
132
+ │ ├── project.yml uses: ../../shared
133
+ │ ├── collections/login.yml
134
+ │ └── environments/local.yml
135
+ ├── services/users/ a project
136
+ │ └── collections/users.yml
137
+ └── shared/ a global project
138
+ ├── project.yml
139
+ └── environments/local.yml
140
+ ```
141
+
142
+ A project reads nothing above its own folder except the global project it names.
143
+
144
+ ### 1.1 `project.yml` and global projects
145
+
146
+ ```yaml
147
+ name: Payments # shown instead of the folder name
148
+ uses: ../../shared # a global project, relative to this one
149
+ vars: # for every collection in the project
150
+ region: eu
151
+ tls:
152
+ ca: # certificate files to trust, besides the system's
153
+ - certs/company-root.pem
154
+ ```
155
+
156
+ The file is optional, and so is every key in it. Any other key is an error.
157
+
158
+ | Key | Type | Default | Meaning |
159
+ | -------- | ---------------------- | ----------- | ----------------------------------------------------- |
160
+ | `name` | string | folder name | Display name. |
161
+ | `uses` | string (relative path) | none | A global project whose files this one shares (below). |
162
+ | `vars` | map of plain values | none | Variables for every collection in the project (§4). |
163
+ | `tls.ca` | list of relative paths | none | Certificate files that requests trust (below). |
164
+
165
+ **`uses`** names a **global project**: an ordinary project whose `project.yml`
166
+ variables, `environments/`, `requests/`, `endpoints/`, `bases/`, `checks/`,
167
+ `settings.yml` and `rules.yml` every project using it shares.
168
+
169
+ - It must be a relative path. An absolute path is refused, since it would only work on
170
+ one machine. Write it with `/`; `\` reads the same.
171
+ - The folder it names must hold a `project.yml`.
172
+ - A global project must not `uses` another: one level, no chains, no loops.
173
+ - An environment in the global project merges under the project's environment of the
174
+ same name, key by key, and the project's values win. An environment only the global
175
+ project has is available too.
176
+ - The global project's `settings.yml` lies under the project's the same way (§1.3), and
177
+ so does its `rules.yml` (§1.4).
178
+ - A global project's own `collections/` is **not** shared: a project using it never
179
+ sees or runs those collections. A global project needs none. It may have one for a
180
+ single purpose: **testing what it shares**. A collection there can `use:` its reusable
181
+ requests, `extends:` its bases and call its checks, against its own environments, so a
182
+ broken shared piece fails in one place, before every project relying on it does.
183
+ Those collections run only when the global project itself does, as a project in
184
+ Accelerator or with `ata` in its folder, such as its own CI job.
185
+
186
+ **`tls.ca`** lists certificate files that requests trust, for a server whose certificate
187
+ a company or local CA signed, or a server's own self-signed certificate. A request
188
+ always trusts:
189
+
190
+ 1. Node's bundled Mozilla roots, and any in `NODE_EXTRA_CA_CERTS`.
191
+ 2. The operating system's trust store: the macOS Keychain, the Windows certificate
192
+ store, or the Linux CA bundle. A CA that IT installed, or that `mkcert -install`
193
+ added, is trusted with nothing written here.
194
+ 3. `tls.ca`: the project's own files, then its global project's.
195
+
196
+ - Each entry is a relative path from the `project.yml` that lists it. An absolute path
197
+ is refused.
198
+ - A file is PEM (one certificate or a bundle) or a single DER certificate, such as a
199
+ `.cer` exported on Windows. A CA's certificate is public and safe to commit. A private
200
+ key never belongs here.
201
+ - A file that is missing or holds no certificate is a problem on the project. Accelerator
202
+ sends nothing from the project until it is fixed, and `ata` will not start.
203
+ - `tls.ca` only adds trust. Host names and expiry are still checked, and verification
204
+ is never turned off.
205
+
206
+ ### 1.2 Portability
207
+
208
+ Projects are shared between macOS, Windows and Linux, and read the same on all three:
209
+
210
+ - Every path written in a file uses `/`. A `\` reads the same, but is never written.
211
+ - **Names match exactly, case included.** macOS and Windows find `requests/Auth/login.yml`
212
+ when the file is `requests/auth/login.yml`; Linux does not, so a project that works on a
213
+ laptop would fail in CI. Every name a file refers to must be spelled as it is on disk: a
214
+ `use:` or `extends:` name, `uses`, a `tls.ca` file, a file a body sends, and an
215
+ environment's file name. One that differs only in case is an error on every platform,
216
+ naming the spelling on disk. The folders and files of §1 are lower case.
217
+ - **No two names in one folder may differ only in case**, such as `Checkout/` and
218
+ `checkout/`: Linux can hold both, but a macOS or Windows checkout only one. This
219
+ applies in `collections/`, `requests/`, `endpoints/`, `bases/`, `environments/` and
220
+ `checks/`. Collection ids are unique ignoring case too (§2).
221
+ - File, folder and environment names must avoid what Windows refuses: the characters
222
+ `< > : " / \ | ? *`, control characters, a trailing dot or space, and the names `CON`,
223
+ `PRN`, `AUX`, `NUL`, `CONIN$`, `CONOUT$`, `COM1`–`COM9` and `LPT1`–`LPT9`, with or
224
+ without an extension. A file with such a name, made on macOS or Linux, is reported.
225
+ - Files should use LF line endings. Accelerator writes LF, and can add a `.gitattributes`
226
+ scoped to the project's own files so that git keeps them LF on every platform. The same
227
+ block keeps `files/` exactly as committed, so a body sends the same bytes everywhere. A
228
+ global project is a project too: its own block covers its `files/`. For files a body
229
+ sends from another folder, add a `-text` line of your own.
230
+ - The process environment is read by exact name on every platform, although Windows
231
+ itself ignores case (§4, §6).
232
+
233
+ ### 1.3 `settings.yml`
234
+
235
+ How `ata` runs the project. It sits beside `collections/`, is committed, and `ata`
236
+ refuses to run without it. The desktop app does not read it.
237
+
238
+ ```yaml
239
+ environmentType: staging # environments/<name>.yml
240
+ limitConcurrency: 4
241
+ timeoutCollection: 3600000
242
+ bail: false
243
+ tags: [smoke]
244
+ notTags: [slow]
245
+ generateJUnitResults: true
246
+ generateJsonResults: true
247
+ generateHtmlResults: true
248
+ autoOpenTestResultHtml: false
249
+ testResultsBasePath: test-results
250
+ ```
251
+
252
+ Every key is optional. **A key not listed here is an error**, so a typo cannot quietly
253
+ run with a default.
254
+
255
+ | Key | Type | Default | Meaning |
256
+ | ------------------------ | ------------ | -------------- | --------------------------------------------------------------------------- |
257
+ | `environmentType` | string | none | The environment to run against, by name. It must exist (§6). |
258
+ | `limitConcurrency` | integer ≥ 1 | `1` | Collections run at once. Steps within a collection always run in order. |
259
+ | `timeoutCollection` | integer ≥ 1 | `3600000` | Stop a collection after this many ms and fail it; its teardown still runs. |
260
+ | `bail` | boolean | `false` | Stop a collection at its first failing step; the rest are reported skipped. |
261
+ | `tags` | list of tags | `[]` | Run only what these select (§2.4). Empty runs everything. |
262
+ | `notTags` | list of tags | `[]` | Leave out what these name (§2.4). |
263
+ | `generateJUnitResults` | boolean | `false` | Write `<testResultsBasePath>/junit/junit.xml`. |
264
+ | `generateJsonResults` | boolean | `false` | Write `<testResultsBasePath>/json/results.json`. |
265
+ | `generateHtmlResults` | boolean | `false` | Write `<testResultsBasePath>/html/summary.html` and a page per collection. |
266
+ | `autoOpenTestResultHtml` | boolean | `false` | Write the HTML report and open it when the run ends. |
267
+ | `testResultsBasePath` | string | `test-results` | Where reports go: relative to the project folder, or absolute. |
268
+
269
+ The results folder is emptied before every run. `ata` refuses to empty one that holds
270
+ files it did not write, or one that holds the project itself. Every report replaces
271
+ each secret's value with `[secret: NAME]` (§6).
272
+
273
+ **From the global project.** A global project (§1.1) can have a `settings.yml` too.
274
+ Every project that `uses` it runs with those settings under its own: a key the
275
+ project's file sets wins, and a key it leaves out comes from the global project's file.
276
+
277
+ - There is no switch to turn this off. A project undoes a shared setting by setting it
278
+ in its own file, back to the default if need be: `environmentType: null`, `tags: []`.
279
+ - A project still needs a `settings.yml` of its own, even an empty one. A global project
280
+ need not have one.
281
+ - A relative `testResultsBasePath` is relative to the project being run, whichever file
282
+ set it.
283
+
284
+ **Overrides.** Each setting can be overridden by an environment variable, then by a
285
+ command-line flag: `ATA_LIMIT_CONCURRENCY=8`, then `--limitConcurrency 8`. A list is
286
+ comma-separated: `--tags smoke,api`. In all, from lowest to highest: the default, the
287
+ global project's `settings.yml`, the project's, the environment variable, the flag.
288
+ `ata get` lists each setting that is not a default, and where it came from.
289
+
290
+ **Running.** `ata` runs from the project folder:
291
+
292
+ | Command | Runs |
293
+ | --------------------- | -------------------------------------------------------------------- |
294
+ | `ata get` | Nothing. It lists what `ata all` would run. |
295
+ | `ata all` | Every collection and step, except those with `exclude: true` (§2.4). |
296
+ | `ata smoke,checkout/` | The collections and folders named, in that order. |
297
+
298
+ `ata get` also reports each `use:` and `extends:` that would stop a run, and each file a
299
+ body names that is in neither the project nor its global project (§2.2), in every
300
+ collection, including those `ata all` leaves out (Appendix A). A file path with
301
+ `{{variables}}` is left to the run. It exits `1` when it finds one, or a broken
302
+ collection, and `0` otherwise.
303
+
304
+ A collection is named by its `id`, or by its place (`checkout/sessions`). A folder is
305
+ named by its name, and `checkout/` names only the folder. `--flag name=value` sets a
306
+ feature flag (§2.9), and `--json` prints the results as JSON. The exit code is `0` when
307
+ everything passed, `1` when something failed, and `2` when `ata` could not run.
308
+
309
+ ### 1.4 `rules.yml`
310
+
311
+ How a project's files are named, laid out and written, so that everyone working on it,
312
+ people and coding agents alike, keeps it consistent. It sits beside `project.yml`, is
313
+ committed, and is optional. `ata lint` checks the project against it. Accelerator, the
314
+ desktop app, marks each file, folder and step that breaks a rule, and will not create or
315
+ rename a file or folder that would.
316
+
317
+ ```yaml
318
+ ids:
319
+ collections: kebab-case # or a pattern, such as '{folder}-[a-z0-9-]+'
320
+ requests: camelCase
321
+ layout:
322
+ folders: required # every collection in a folder of collections/
323
+ folderNames: [accounts, payments, smoke, payments-errors]
324
+ kinds:
325
+ negative: # tagged negative: in a folder ending in -errors, which hold no other
326
+ tags: [negative]
327
+ folders: .*-errors
328
+ maxSteps: 30
329
+ steps:
330
+ names: required
331
+ url: ^\{\{baseUrl\}\} # no host written out
332
+ docs:
333
+ collections: required
334
+ tags:
335
+ allowed: [smoke, regression, slow]
336
+ tests:
337
+ only: [ata] # tests call the ata.* functions, and nothing else
338
+ statusCode: required
339
+ guide: |
340
+ Name each step for what it proves. Logins go through `requests/login`.
341
+ ```
342
+
343
+ **Rules never change a run.** A file that breaks one opens, sends and runs as before.
344
+ `ata lint` reports it, and a CI job running `ata lint` fails on it. Accelerator marks it, and
345
+ flags what `tests.only` does not allow as a script is typed.
346
+
347
+ | Rule | Value | What it checks |
348
+ | -------------------- | ---------------------- | ------------------------------------------------------------------------------- |
349
+ | `ids.collections` | style or pattern | Each collection's id, its file name (§2). |
350
+ | `ids.requests` | style or pattern | Each reusable requests file's id (§2.5). |
351
+ | `ids.bases` | style or pattern | Each base collection's id (§2.7). |
352
+ | `ids.endpoints` | style or pattern | Each endpoints file's id (§2.6). |
353
+ | `layout.folders` | `required` | Every collection sits in a folder of `collections/`, none at its top. |
354
+ | `layout.folderNames` | list, style or pattern | Each folder of `collections/`: one of the list, or following the format. |
355
+ | `layout.kinds` | map of kinds (below) | Each kind of collection sits in its own folders, and they hold no other. |
356
+ | `layout.maxSteps` | integer ≥ 1 | The most steps a collection has in `steps`; `setup` and `teardown` aside. |
357
+ | `layout.bases` | `none` | No base collections: no file in `bases/`, and no `extends:` (below). |
358
+ | `layout.checks` | `none` | No check files: no `.js` file in `checks/`, and no use of `checks` (below). |
359
+ | `steps.names` | `required` | Every step has a `name`, and no other step of its file has the same one. |
360
+ | `steps.url` | pattern | Every request step's URL, as written, matches it (below). |
361
+ | `docs.collections` | `required` | Every collection has `docs`. |
362
+ | `docs.requests` | `required` | Every reusable requests file has `docs`. |
363
+ | `docs.steps` | `required` | Every step of a collection or reusable requests file has `docs`, in every list. |
364
+ | `tags.allowed` | list of tags | Every tag on a collection or a step is one of these. |
365
+ | `tags.collections` | `required` | Every collection has `tags` of its own. |
366
+ | `tests.only` | list (below) | What a `tests` script may call. |
367
+ | `tests.everyStep` | `required` | Every step that sends a request or reads a connection is checked (below). |
368
+ | `tests.statusCode` | `required` | Every request step's checks include its status code (below). |
369
+
370
+ - Any other group or rule is an error, as is a value of the wrong type.
371
+ - A rule that takes `required` also takes `optional`, which turns it off, and one that
372
+ takes `none` takes `allowed`.
373
+ - A **style** is one of `kebab-case` (`create-user`), `snake_case` (`create_user`),
374
+ `camelCase` (`createUser`) and `PascalCase` (`CreateUser`). Lower case, in the first two,
375
+ includes digits: `404-handling` is kebab-case.
376
+ - A **pattern** is a JavaScript regular expression that the whole name must match, as if
377
+ it began with `^` and ended with `$`. `{folder}` in it stands for the folder the file is
378
+ in, so `{folder}-[a-z0-9-]+` asks for `collections/payments/payments-refunds.yml`. A
379
+ value that is not a style and has none of a pattern's characters (`^`, `$`, `.`, `*`,
380
+ `+`, `?`, `(`, `)`, `[`, `]`, `{`, `}`, `|` or a backslash) is an error, so a
381
+ misspelled style is not taken for a pattern.
382
+ - Rules check the project's own files, in `collections/`, `requests/`, `bases/` and
383
+ `endpoints/`, and `layout.checks` its `checks/`. A global project's files follow the
384
+ global project's `rules.yml`, checked when `ata lint` runs in its folder.
385
+ - `steps.*`, `docs.steps`, `tests.everyStep` and `tests.statusCode` are about the steps of
386
+ collections and reusable requests files. An endpoint is a method and a path pattern,
387
+ not a step that runs, so they leave endpoints files alone.
388
+
389
+ **`steps.url`** is a JavaScript regular expression that each request step's URL, as
390
+ written with its `{{variables}}`, must match. Unlike an id pattern it is not anchored:
391
+ `^\{\{baseUrl\}\}` asks that every URL start with `{{baseUrl}}`, so no host is written
392
+ out in a step. A use step and a step reading a connection have no URL of their own.
393
+
394
+ **`tests.everyStep` and `tests.statusCode`** count every script that runs after a step's
395
+ response, wherever it is written: the step's own `tests`, its file's, its base
396
+ collection's (§2.7) and its endpoint's (§2.6), unless the step has `base: false`. With
397
+ `tests.statusCode`, one of them calls `ata.expectResponseStatusCodeToBe`, in any branch,
398
+ or calls a check function (§5) whose own code does. A use step is left to its reusable
399
+ requests file, whose steps are checked there. A step reading a connection needs tests
400
+ under `tests.everyStep`, but has no status code of its own to check.
401
+
402
+ **`tests.only`** lists what a `tests` script may call. It must list `ata`:
403
+
404
+ | Entry | Allows |
405
+ | ---------- | --------------------------------------------------------- |
406
+ | `ata` | The `ata.*` functions (§5), except `ata.test`. |
407
+ | `ata.test` | Named checks of your own, with any code inside them (§5). |
408
+ | `checks` | The project's check files, `checks.<file>.<function>()`. |
409
+ | `console` | `console.log()` and the rest. |
410
+
411
+ With `only: [ata]`, a `tests` script holds calls to `ata.*` functions and nothing else:
412
+
413
+ - **Every argument is a value**: a string, number, `true`, `false`, `null` or regular
414
+ expression, a template string, a list or object of values, or `new RegExp(…)` of
415
+ values. It may read `res`, `req`, `params`, `endpoint` and `item` (§5), call another
416
+ `ata.*` function such as `ata.get('id')`, or `JSON.parse` a value, since `ata.set` keeps
417
+ a list or object as JSON text.
418
+ - **An `if` may choose by feature flag** (§2.9): its condition is `ata.flag()` calls
419
+ compared with values, joined by `!`, `&&` and `||`, and what it holds follows the same
420
+ rules.
421
+ - **Nothing else**: no variables, loops, functions of your own, `assert`, `checks.*` or
422
+ `ata.test`. This is checked on the script's syntax, so a call cannot be hidden by
423
+ renaming it (`const c = checks`).
424
+
425
+ ```yaml
426
+ tests: |
427
+ ata.expectResponseStatusCodeToBe(200)
428
+ ata.expectResponseBodyToHaveProperty('owner', ata.get('userId'))
429
+ ata.set('nextPage', res.body.links.next)
430
+ if (ata.flag('newCheckout') === true) {
431
+ ata.expectResponseBodyToHaveProperty('version', 2)
432
+ }
433
+ ```
434
+
435
+ `tests.only` checks every `tests` script: a collection's own and each of its steps', and
436
+ those of reusable requests files, base collections and endpoints files.
437
+ `before.script` is not checked.
438
+
439
+ **`layout.kinds`** keeps each kind of collection in its own folders: negative tests in
440
+ the `-errors` folders, say, and nothing else there.
441
+
442
+ ```yaml
443
+ layout:
444
+ kinds:
445
+ negative:
446
+ tags: [negative] # a collection tagged negative is one
447
+ id: .*_N # and so is one whose id ends in _N
448
+ folders: .*-errors # it goes in a folder ending in -errors, and those hold no other
449
+ ```
450
+
451
+ - Each kind has a name of your choosing, used in its findings, and three keys:
452
+ - **`tags`**, a list: a collection with any of these tags of its own is one. A step's
453
+ tags do not count.
454
+ - **`id`**, a style or pattern: so is a collection whose id follows it. Like an id
455
+ pattern, it matches the whole id, so `.*_N` is an id ending in `_N`.
456
+ - **`folders`**, required: a list of folder names, a style or a pattern, as
457
+ `layout.folderNames` takes. A pattern matches the whole name, so `.*-errors` is a
458
+ folder ending in `-errors`, while `-errors$` matches only a folder called `-errors`.
459
+ - A kind needs `tags`, `id` or both. With both, either makes a collection one.
460
+ - A kind's `id` does not loosen `ids.collections`. An id ending in `_N` is neither
461
+ kebab-case nor snake_case, so a project with both sets a pattern that allows it, such
462
+ as `ids: { collections: '[a-z0-9_]+(_N)?' }`.
463
+ - **One of a kind outside its folders** is a finding, at the tag that makes it one, or
464
+ at its `id` when the id does. A collection at the top of `collections/` is in no folder.
465
+ - **One not of the kind inside them** is a finding too. So collections without the tag
466
+ or the id, the positive tests here, stay out of the `-errors` folders.
467
+ - Each kind is checked on its own, so a project can have several, such as `slow` in
468
+ `slow/` beside `negative` in `.*-errors`.
469
+ - Accelerator will not rename, move or copy a collection, or rename a folder, where it
470
+ would break this. A new collection may be made in a kind's folders when the kind has
471
+ `tags`, since it can only be tagged once it exists. Until it is, it is a finding.
472
+ - A project's `layout.kinds` replaces its global project's whole, as every rule's value
473
+ does. `null` turns it off.
474
+
475
+ **`layout.bases` and `layout.checks`** keep a project to what its own files say, with no
476
+ base collection (§2.7) or check file (§5) adding to them:
477
+
478
+ ```yaml
479
+ layout:
480
+ bases: none
481
+ checks: none
482
+ ```
483
+
484
+ - **`layout.bases: none`**: each file in `bases/` is a finding, and so is every
485
+ `extends:`, whether it names one of the project's bases or one of its global project's
486
+ (`global:name`). Accelerator offers no new base collection, and no base to choose for a
487
+ collection that has none.
488
+ - **`layout.checks: none`**: each `.js` file in `checks/` is a finding, and so is every
489
+ use of `checks` in a script: `before.script` and `tests`, the file's own and each
490
+ step's, in every file the rules check. A use is any reference to `checks`, so
491
+ `const c = checks` is one too. It is a finding whichever project the check file is in.
492
+ - With `tests.only` too, a `checks.*` call in `tests` breaks both rules, but it is
493
+ reported once, under `layout.checks`.
494
+ - To keep every project that shares a global project from them, set them in the global
495
+ project's `rules.yml`. A project can still turn either off in its own, with `allowed`
496
+ or `null`, and that shows in its diff and in `ata rules`.
497
+
498
+ **`guide`** is Markdown, for what no rule can check: how steps are named, which request
499
+ set a login goes through, what a collection is for. `ata rules` prints it after the
500
+ rules, and Accelerator shows it in Project settings. Each file's guide is read, the global
501
+ project's first, so a project adds to a shared guide rather than replacing it.
502
+
503
+ **From the global project.** A global project (§1.1) can have a `rules.yml` too, and it
504
+ lies under the project's, rule by rule. There is no switch to turn this off: a project
505
+ changes a shared rule by setting it in its own file, and turns it off with `null`, or
506
+ `optional` for a `required` rule. A group set to `null`, such as `tags: null`, turns off
507
+ every rule in it. Neither file is required.
508
+
509
+ **Checking.** From a project folder, or a global project's:
510
+
511
+ | Command | Does |
512
+ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
513
+ | `ata lint` | Checks every file against the rules, and prints each finding at its file and line, with its rule and the file the rule came from. Exits `1` on a finding. |
514
+ | `ata rules` | Lists each rule set, its value, the file it came from and what it means, rules turned off included, then the guide. |
515
+
516
+ - Neither needs `settings.yml` or runs anything, and both take `--json`.
517
+ - A file that will not parse is a finding, since no rule could be checked in it.
518
+ - Both exit `2` when a rule is not valid, or `project.yml` names a global project that
519
+ cannot be used: its rules would quietly go unchecked.
520
+ - A coding agent can run `ata rules` to learn a project's conventions before writing, and
521
+ `ata lint --json` after, to fix what it finds. Accelerator offers to add a section saying so
522
+ to the project's `AGENTS.md`, which coding agents read; nothing is written until asked.
523
+
524
+ ---
525
+
526
+ ## 2. Collection file
527
+
528
+ A **collection** is one file holding an ordered list of requests, called **steps**. Its
529
+ steps run in list order and share one variable scope, so what one step captures, the
530
+ next can use.
531
+
532
+ ```yaml
533
+ # collections/checkout.yml
534
+ id: checkout
535
+ docs: |
536
+ Create a session, capture it, then read it back.
537
+ tags: [smoke]
538
+ headers: # sent with every step
539
+ Accept: application/json
540
+ settings:
541
+ timeout: 10000
542
+ vars:
543
+ apiVersion: '2'
544
+
545
+ steps:
546
+ - name: create session
547
+ POST: '{{baseUrl}}/v{{apiVersion}}/sessions'
548
+ body:
549
+ json: |
550
+ { "amount": 1200 }
551
+ tests: |
552
+ ata.expectResponseStatusCodeToBe(201)
553
+ ata.expectResponseBodyToHaveProperty('id', 'sessionId', 'setAsCollectionVariable')
554
+
555
+ - name: read it back
556
+ GET: '{{baseUrl}}/v{{apiVersion}}/sessions/{{sessionId}}'
557
+ tests: |
558
+ ata.expectResponseStatusCodeToBe(200)
559
+ ata.expectResponseBodyToHaveProperty('amount', 1200)
560
+ ```
561
+
562
+ ### Collection keys
563
+
564
+ Any key not listed here is an error.
565
+
566
+ | Key | Type | Required | Default | Meaning |
567
+ | ---------- | ------------------- | -------- | ------- | ----------------------------------------------------------------------------------------- |
568
+ | `id` | string | **yes** | | The file name without `.yml` (below). |
569
+ | `steps` | list of steps | no | `[]` | The requests, in run order (§2.1). |
570
+ | `setup` | list of steps | no | | Run once before `steps`; what it sets lasts the run (§2.10). |
571
+ | `teardown` | list of steps | no | | Run once after the rest, even when a step failed (§2.10). |
572
+ | `docs` | string | no | | Markdown (below). |
573
+ | `tags` | list of tags | no | | Tags that select the whole collection (§2.4). |
574
+ | `stepTags` | boolean | no | `false` | `true` lets steps carry their own tags (§2.4). |
575
+ | `exclude` | boolean | no | `false` | `true` leaves it out of group runs (§2.4). |
576
+ | `flags` | map | no | | Feature flags the whole collection needs (§2.9). |
577
+ | `headers` | map | no | | Sent with every step; a step's own header of the same name wins (§2.3). |
578
+ | `settings` | map | no | | Defaults for every step (§2.3). |
579
+ | `vars` | map of plain values | no | | Collection variables (§4). |
580
+ | `before` | map | no | | `script:` run before every step (§5). |
581
+ | `tests` | string | no | | JavaScript run after every step, before the step's own (§5). |
582
+ | `extends` | string | no | | A base collection to build on (§2.7). |
583
+ | `params` | map | no | | The inputs it takes as a reusable requests file (§2.5). Only in `requests/`, in practice. |
584
+
585
+ A collection has no `name` key. Its `id` is its name, and a file with `name:` is
586
+ rejected with a message saying so.
587
+
588
+ ### `id`
589
+
590
+ Every collection file must say its **id**, which is its file name without `.yml`,
591
+ exactly. `collections/checkout/sessions.yml` starts `id: sessions`.
592
+
593
+ - **It must match the file name.** A file whose `id` is missing or different is a
594
+ broken collection. Renaming a file means changing its `id` too.
595
+ - **It must be unique in its home, ignoring case.** No two files in a project's
596
+ `collections/` may share an id, including files in different folders of it; the
597
+ same holds for `requests/`, `bases/` and `endpoints/`. Case is ignored because macOS
598
+ and Windows file systems ignore it. Every file sharing an id is broken.
599
+ - **It is letters, digits and `- _ .`, starting with a letter or digit**:
600
+ `^[A-Za-z0-9][A-Za-z0-9._-]*$`. It is also typed on a command line.
601
+
602
+ Because an id is unique, it names the collection everywhere on its own: in the app, on
603
+ `ata`'s command line and in reports.
604
+
605
+ ### `docs`
606
+
607
+ A collection's or a step's `docs` is Markdown, shown in the desktop app and in `ata`'s
608
+ HTML report: headings, paragraphs, lists, tables, fenced code, block quotes, rules, and
609
+ inline code, emphasis and links (http and https only). HTML is shown as the text it is,
610
+ never rendered. Anything else stays the text it was written as.
611
+
612
+ GitHub's callouts are a block quote whose first line is the callout's name, and are shown
613
+ with its colour, icon and title:
614
+
615
+ ```markdown
616
+ > [!WARNING]
617
+ > The cancel step needs the `orders:write` scope.
618
+ ```
619
+
620
+ The names are `NOTE`, `TIP`, `IMPORTANT`, `WARNING` and `CAUTION`, in any case.
621
+
622
+ Some text is coloured:
623
+
624
+ - `{{variables}}` are marked, in code or not, as the app's URL bar marks them.
625
+ - Inline code that starts with a method, such as `` `GET /orders` ``, shows the method in
626
+ its colour. Inline code that is a status code, such as `` `201` `` or
627
+ `` `404 Not Found` `` or `` `5xx` ``, is shown in its class's colour.
628
+ - Fenced code is coloured when its fence names `json` or `xml`. The app also colours
629
+ `js` and `ts`.
630
+
631
+ ### 2.1 Steps
632
+
633
+ **List order is run order.** A step is a request: it carries **exactly one HTTP method
634
+ key**, in capitals, whose value is the URL as a string.
635
+
636
+ ```yaml
637
+ - name: create session
638
+ POST: '{{baseUrl}}/sessions?source=api'
639
+ ```
640
+
641
+ Methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`.
642
+
643
+ | Key | Type | Required | Meaning |
644
+ | ------------ | ------------ | -------- | ------------------------------------------------------------------------------- |
645
+ | `<METHOD>` | string | **yes** | The URL, query string included. |
646
+ | `name` | string | no | Display name. Defaults to the method and URL. |
647
+ | `headers` | map | no | Over the collection's headers (§2.3). |
648
+ | `body` | map | no | Exactly one kind of body (§2.2). |
649
+ | `settings` | map | no | Over the collection's settings (§2.3). |
650
+ | `before` | map | no | `script:` run before the request (§5). |
651
+ | `tests` | string | no | The checks, as JavaScript: calls on `ata` (§3) and any other code (§5). |
652
+ | `tags` | list of tags | no | The step's own tags. Only with `stepTags: true` on the collection (§2.4). |
653
+ | `exclude` | boolean | no | `true` leaves it out of group runs. Only on a step of `steps` (§2.4). |
654
+ | `flags` | map | no | Feature flags the step needs (§2.9). |
655
+ | `forEach` | string | no | In setup or teardown: send the request once for each item of a list (below). |
656
+ | `useTests` | `true` | no | In a reusable requests file: the use step's `tests` check this response (§2.5). |
657
+ | `connection` | string | no | Keep the event stream this request opens as a connection (§2.11). |
658
+ | `base` | `false` | no | `false` leaves the step's endpoint base out (§2.6). |
659
+ | `docs` | string | no | Markdown (§2, `docs`). |
660
+ | `wait` | int, string | no | Wait this many milliseconds instead of sending a request (below). |
661
+
662
+ - A step with two method keys is an error, and so is a step with none, unless it is a
663
+ use step, reads a connection or waits.
664
+ - **The URL is authoritative, query string included.** There is no separate block of
665
+ query parameters. Editors show a parameter table as a view over the URL.
666
+ - A step may instead run reusable requests with `use:` (§2.5). A use step has no
667
+ method key.
668
+ - A step may also read a connection, an event stream an earlier step keeps open, with
669
+ `connection:` and no method key (§2.11).
670
+ - A step may only wait, with `wait:` and no method key (below).
671
+ - **Any other key is an error**, so a misspelled key such as `heders:` fails at once
672
+ rather than being ignored. A method key in lower case (`get:`) is reported as such.
673
+
674
+ **`forEach`**, on a setup or teardown step (§2.10), sends the request once for each item
675
+ of a list, each reported as its own result, `teardown › remove grant (item 2 of 3)`:
676
+
677
+ ```yaml
678
+ teardown:
679
+ - name: remove grant
680
+ DELETE: '{{baseUrl}}/grants/{{item}}'
681
+ forEach: '{{grantedRoots}}' # a variable holding ["r1", "r2"], or a list written in place
682
+ ```
683
+
684
+ - It is for a list of things to prepare or clean up, such as every grant the rows made.
685
+ A step in `steps` cannot have it: a test sends each of its requests once, and a data
686
+ file (§2.8) repeats the steps for each row. Nor can a reusable requests file's steps.
687
+ - Its `{{variables}}` resolve when the step runs, so an earlier step can make the list:
688
+ `ata.set('grantedRoots', ids)` in setup, or with `{ scope: 'run' }` in a row (§2.10).
689
+ The result must be a JSON array.
690
+ - Each item is `{{item}}` in the request and `item` in its scripts. An item that is an
691
+ object or a list is its JSON in the request.
692
+ - An empty list skips the step. Anything that is not a list fails it before anything
693
+ is sent.
694
+ - A use step cannot have `forEach`.
695
+
696
+ **`wait`** makes a step that sends nothing: it waits, then the next step runs. It is for
697
+ something the API does after it answers, such as a search index catching up:
698
+
699
+ ```yaml
700
+ steps:
701
+ - name: create order
702
+ POST: '{{baseUrl}}/orders'
703
+ - wait: 2000 # milliseconds; reports call it "wait 2 s"
704
+ - name: order is searchable
705
+ GET: '{{baseUrl}}/search?q={{orderId}}'
706
+ ```
707
+
708
+ - It is a whole number of milliseconds, from 0 to 600000 (10 minutes). A fixed wait any
709
+ longer is nearly always waiting for something to be ready, which a step that checks
710
+ for it does better.
711
+ - It may be a string of `{{variables}}`, resolved when the step runs, such as
712
+ `wait: '{{retryAfterMs}}'` with a value an earlier step's `tests` set with `ata.set`.
713
+ What it comes to must be a whole number from 0 to 600000, or the step fails. A string
714
+ that holds no `{{variable}}`, such as `2s`, is an error.
715
+ - A wait step has `wait`, and may have `name`, `docs`, `flags`, `tags` and `exclude` (§2.4,
716
+ §2.9).
717
+ It has no method key, `use`, `connection`, `headers`, `body`, `settings`, `before`,
718
+ `tests`, `forEach`, `useTests` or `base`. Its name defaults to how long it waits:
719
+ `wait 2 s`, `wait 500 ms`, `wait {{retryAfterMs}}`.
720
+ - No script runs for it, the collection's included: there is no request or response.
721
+ Its feature flags, `ata.skipRest`, `bail`, tags and `exclude` apply as to any step.
722
+ - Its result has the method `WAIT`, no response and no checks, and passes once it has
723
+ waited. A cancel, or `timeoutCollection`, stops it at once, as a failure; teardown
724
+ then runs as §2.10 says.
725
+ - It can be a step of `steps`, `setup` and `teardown`, and of a reusable requests file
726
+ (§2.5). An endpoints file holds none (§2.6).
727
+
728
+ ### 2.2 `body`
729
+
730
+ A body declares **exactly one** of these keys. Omit `body` for no body.
731
+
732
+ ```text
733
+ body: { json: '{ "a": 1 }' } # Content-Type: application/json
734
+ body: { xml: '<a/>' } # application/xml
735
+ body: { text: hello } # text/plain
736
+ body: { form: { a: '1', b: two } } # application/x-www-form-urlencoded
737
+ body: { multipart: { … } } # multipart/form-data (below)
738
+ body: { graphql: { query: '…', variables: { … } } } # application/json
739
+ body: { file: files/order.json } # the file as it is, typed by its extension
740
+ ```
741
+
742
+ | Key | Type | Sent as |
743
+ | ----------- | ------------------------------------ | ---------------------------------------------------------- |
744
+ | `json` | string | The text, as written. It must be a string, not a YAML map. |
745
+ | `xml` | string | The text. |
746
+ | `text` | string | The text. |
747
+ | `form` | map of strings | URL-encoded, names and values resolved first. |
748
+ | `multipart` | map of fields (below) | `multipart/form-data`. |
749
+ | `graphql` | `{ query: string, variables?: map }` | `{"query": …, "variables": …}` as JSON. |
750
+ | `file` | string (relative path) | The file's bytes. |
751
+
752
+ - The implied `Content-Type` is added only when `headers` does not declare one.
753
+ - `{{variables}}` resolve in every kind of body. A form's values are resolved and then
754
+ encoded, so a value may hold `&` or `=`.
755
+ - A number in `form` or `multipart` must be quoted: `count: '3'`.
756
+
757
+ Write JSON as a block, so it needs no escaping:
758
+
759
+ ```yaml
760
+ body:
761
+ json: |
762
+ {
763
+ "user": "{{username}}",
764
+ "amount": 1200
765
+ }
766
+ ```
767
+
768
+ **`multipart`** is keyed by field name, in order:
769
+
770
+ ```yaml
771
+ body:
772
+ multipart:
773
+ description: A photo of {{name}} # text
774
+ metadata: # text with a Content-Type of its own
775
+ value: '{"album": "{{album}}"}'
776
+ contentType: application/json
777
+ avatar: # a file from the project folder
778
+ file: files/avatar.png
779
+ contentType: image/png # optional: else from the extension, else application/octet-stream
780
+ filename: me.png # optional: else the file's own name
781
+ tags: [red, blue] # a list sends the name once for each
782
+ ```
783
+
784
+ | A field is | Sent as |
785
+ | ----------------------------------- | ------------------------------------------------ |
786
+ | a string | A text part. |
787
+ | `{ value, contentType? }` | A text part with its own `Content-Type`. |
788
+ | `{ file, contentType?, filename? }` | A file part. |
789
+ | a list of the above | One part for each item, all with the field name. |
790
+
791
+ `filename: ''` sends an empty name, as a browser does when no file is chosen. Pair it
792
+ with an empty file for the whole of that request.
793
+
794
+ The `Content-Type` is `multipart/form-data` with a boundary chosen for the request. A
795
+ declared multipart type without a boundary, such as `multipart/mixed`, gets one added.
796
+ A declared boundary is used as written.
797
+
798
+ **Files**, in `body.file` and a multipart part's `file`, are read from the folder of
799
+ the project the step belongs to. For a step of a reusable requests file, that is the
800
+ file's own project, which may be a global one. The path is the same wherever the
801
+ collection sits inside `collections/`.
802
+
803
+ A file that is not there is looked for in the global project (§1.1), as a reusable
804
+ requests file or a base collection is, so projects can share one copy of a file. A
805
+ project's own file of the same path wins. `global:` before the path reads only the
806
+ global project's:
807
+
808
+ ```yaml
809
+ body:
810
+ multipart:
811
+ avatar: { file: files/test-png.png } # the project's, else the global project's
812
+ terms: { file: global:files/terms.pdf } # the global project's only
813
+ ```
814
+
815
+ - It must be a relative path, written with `/`. An absolute path is refused.
816
+ - A path out of the project folder, such as `../other/files/a.png`, is read from where
817
+ it points, with no global project to fall back to. A `global:` path stays inside the
818
+ global project folder, and is an error in a project that uses none.
819
+ - `{{variables}}` resolve in the path, so a data file row (§2.8) can choose the file.
820
+ - A file is sent byte for byte. `{{…}}` inside it is not a variable.
821
+ - A file that cannot be read stops the step before anything is sent, in its own `body`
822
+ phase, naming the file and each place it was looked for.
823
+ - Where a request is shown (results, reports, `req.body`), a file's bytes appear as
824
+ `‹file files/avatar.png, 1234 bytes›`, and a global project's file as
825
+ `‹file global:files/terms.pdf, 5254 bytes›`, however the step named it.
826
+
827
+ ### 2.3 `settings` and `headers`
828
+
829
+ **`settings`** on a collection apply to every step; a step's own settings merge over
830
+ them. Any other key is an error.
831
+
832
+ | Key | Type | Default | Meaning |
833
+ | ----------------- | -------------------- | ------- | ------------------------------------------------------------------------------------------------------ |
834
+ | `timeout` | number ≥ 0 | `0` | Milliseconds for the whole request; `0` means no limit. For an event stream, until its headers arrive. |
835
+ | `followRedirects` | boolean | `true` | Follow 3xx responses. |
836
+ | `maxRedirects` | integer ≥ 0 | `5` | The most redirects followed. |
837
+ | `encodeUrl` | boolean | `true` | Percent-encode what a hand-typed URL left raw, before sending. |
838
+ | `httpVersion` | `auto`, `1.1` or `2` | `auto` | Which HTTP version to send with; see below. |
839
+ | `maxEvents` | integer ≥ 0 | `0` | Stop reading an event stream after this many events; `0` means no limit. |
840
+ | `streamTimeout` | number ≥ 0 | `0` | Stop reading an event stream this many milliseconds after its headers; `0` means no limit. |
841
+ | `untilEvent` | string | none | Stop reading an event stream after the first event of this name, its `event:` line. |
842
+ | `untilTestsPass` | boolean | `true` | Stop reading an event stream once the step's tests pass, tried after each event; see below. |
843
+
844
+ **HTTP versions.** `httpVersion` takes `2` and `1.1` written plain or quoted.
845
+
846
+ - `auto` uses HTTP/2 when an https server picks it, as most do, and HTTP/1.1 otherwise.
847
+ An http:// URL gets HTTP/1.1.
848
+ - `1.1` never offers HTTP/2.
849
+ - `2` is HTTP/2 or an error. An https server that answers with HTTP/1.1 fails the step
850
+ with `NO_HTTP2`. To an http:// URL, HTTP/2 is spoken in the clear from the start (h2c),
851
+ which a server without it cannot follow.
852
+ - HTTP/3 is not supported.
853
+
854
+ Every result says which HTTP version the request went out on, and whether it went on a
855
+ new connection or on one kept open from an earlier request. The same holds for a request
856
+ that fails once its headers are sent, along with how long it ran before it failed. A
857
+ request dropped on a reused connection points at a server or proxy that closes idle
858
+ connections sooner than the client expects.
859
+
860
+ **Event streams.** A response whose `Content-Type` is `text/event-stream` (Server-Sent
861
+ Events) is read as it arrives, whatever the method, and checked as a list of events
862
+ (§3). Nothing in the step marks it as a stream. Reading stops at whichever comes first:
863
+
864
+ - the server closes the stream
865
+ - `maxEvents` events have arrived
866
+ - `streamTimeout` milliseconds have passed since its headers arrived
867
+ - the first event named `untilEvent` has arrived
868
+ - the step's tests pass on the events so far, with `untilTestsPass` (below)
869
+ - 1,000 events or 10 MB, a limit no setting lifts
870
+
871
+ ```yaml
872
+ - name: price stream
873
+ GET: '{{baseUrl}}/prices/stream?symbol=ACME'
874
+ headers:
875
+ Accept: text/event-stream
876
+ settings:
877
+ maxEvents: 3
878
+ streamTimeout: 5000
879
+ tests: |
880
+ ata.expectResponseStatusCodeToBe(200)
881
+ ata.expectResponseBodyToHaveProperty('[0].event', 'subscribed')
882
+ ata.expectResponseBodyToHaveProperty('[1].data.price', 100)
883
+ ```
884
+
885
+ - Each way of stopping is a normal end, not an error, and the checks run on the events
886
+ that arrived. A check that wanted more fails with the path it missed.
887
+ - Stopped at a number of events, the body ends with the last one.
888
+ - Cancel, or a connection that drops mid-stream, is an error, as for any request.
889
+ - `maxEvents`, `streamTimeout`, `untilEvent` and `untilTestsPass` do nothing to any
890
+ other response.
891
+ - In the desktop app, the events show as they arrive, and Stop ends the reading as
892
+ these do, so the checks run on what came.
893
+ - A step that keeps its stream open for later steps is a connection (§2.11).
894
+
895
+ **Waiting on the tests.** With `untilTestsPass`, on unless set to `false`, the step's
896
+ tests are tried after each event on the events so far, and reading stops at the first
897
+ event they all pass on. That is how to wait for one event among many of the same name,
898
+ picked by what it holds:
899
+
900
+ ```yaml
901
+ - name: job finishes
902
+ GET: '{{baseUrl}}/jobs/{{jobId}}/events'
903
+ settings:
904
+ streamTimeout: 30000 # fail if it has not finished in 30 s
905
+ tests: |
906
+ ata.test('done', () => assert.equal(res.body.at(-1).data.state, 'done'))
907
+ ```
908
+
909
+ - Reading stops on the event that made the tests pass, so that event is the last one,
910
+ `res.body.at(-1)`.
911
+ - The tests pass when they make at least one check and every check passes. Tests that
912
+ never look at the events are not waited on, as they would pass the same on every
913
+ event: only a body check, strict validation, or reading `res.body`, `res.text` or
914
+ `res.stream` looks at them. A step whose tests check only the status is read as if
915
+ `untilTestsPass` were off.
916
+ - A try changes nothing. What it sets, captures or logs is dropped, and so is an
917
+ `ata.skipRest`. Once the reading stops, the tests run as usual on the events it kept,
918
+ and only that run counts.
919
+ - Anything else that stops the reading first ends it as usual, and the tests fail on
920
+ what came. Set `streamTimeout`, or a stream that stays open waits for ever for an event
921
+ that never comes.
922
+ - Every try runs the tests in full, so a slow `ata.test` slows each event.
923
+ - Turn it off for a check that must see the whole stream, such as no `error` event in
924
+ 5 seconds. With it on, such a check passes on the first event, and the reading stops
925
+ there.
926
+
927
+ **Resuming.** A stream is never reconnected: the server closing it ends the step. To
928
+ test that a stream resumes, capture the last event's id and send it as `Last-Event-ID`
929
+ in a later step:
930
+
931
+ ```yaml
932
+ - name: first part
933
+ GET: '{{baseUrl}}/feed'
934
+ tests: |
935
+ ata.set('lastId', res.body.at(-1).id)
936
+ - name: the rest
937
+ GET: '{{baseUrl}}/feed'
938
+ headers:
939
+ Last-Event-ID: '{{lastId}}'
940
+ tests: |
941
+ ata.expectResponseBodyToHaveProperty('[0].id', '4')
942
+ ```
943
+
944
+ **`headers`** is a map from header name to one of:
945
+
946
+ ```yaml
947
+ headers:
948
+ Accept: application/json # a value
949
+ X-Forwarded-For: [10.0.0.1, 10.0.0.2] # a list: sent once per value
950
+ X-Debug: { value: '1', enabled: false, description: turn on to trace } # the long form
951
+ X-Route: [blue, { value: green, enabled: false }] # a list's values can take the long form
952
+ ```
953
+
954
+ - **A header sent more than once is a list**, sent once per value in order. A YAML map
955
+ cannot hold the same key twice, so writing the header twice is an error that says to
956
+ use a list. Names are compared ignoring case, so the desktop app saves rows named
957
+ `X-Tag` and `x-tag` as one list, under the spelling written first.
958
+ - A collection's headers are sent with every step. A step header of the same name,
959
+ compared case-insensitively as HTTP does, replaces the collection's for that step.
960
+ - A header with `enabled: false` is not sent. On a step it replaces nothing, so
961
+ switching it off brings back the collection's header. In a list, a value with
962
+ `enabled: false` is left out and the others are sent. A list with every value
963
+ switched off is not sent, and on a step it replaces nothing.
964
+
965
+ ### 2.4 `tags`, `stepTags` and `exclude`
966
+
967
+ Tags pick what to run. **A collection's `tags` select the whole collection**: every
968
+ step, in order, sharing one variable scope.
969
+
970
+ ```yaml
971
+ id: checkout
972
+ tags: [api, payments] # a run for api or payments runs every step here
973
+ ```
974
+
975
+ Some collections have steps that stand alone. There, `stepTags: true` lets each step
976
+ carry its own tags, so a run can pick single steps:
977
+
978
+ ```yaml
979
+ id: status-codes
980
+ stepTags: true
981
+ steps:
982
+ - name: not found
983
+ GET: '{{baseUrl}}/status/404'
984
+ tags: [smoke, errors]
985
+ ```
986
+
987
+ - **A step must not have `tags` unless its collection has `stepTags: true`.** Most
988
+ collections are a flow, where later steps use what earlier ones set, and running part
989
+ of one fails for reasons that have nothing to do with the API.
990
+ - A tag is letters, digits and `- _ . :` with no spaces (`^[A-Za-z0-9._:-]+$`). Tags are
991
+ case-sensitive and have no prefix. A tag is for grouping only: a feature flag is not
992
+ a tag (§2.9).
993
+
994
+ **Selecting** with `tags`: a collection whose own tags match runs whole. Otherwise, with
995
+ `stepTags: true`, its steps whose tags match run, in order. Otherwise nothing in it runs.
996
+
997
+ **Leaving out** with `notTags`: a collection whose own tags match runs nothing, whatever
998
+ else selects it. With `stepTags: true`, its steps whose tags match are dropped from what
999
+ was selected.
1000
+
1001
+ **`exclude: true`** leaves a collection out of group runs: `ata all`, with or without
1002
+ tags, and a folder named to `ata`. Named on its own, it still runs. Use it for work in
1003
+ progress, a manual-only collection, or one waiting on a fix. `ata` lists what it left
1004
+ out, so a suite never shrinks without saying so.
1005
+
1006
+ **`exclude: true` on a step** leaves just that step out of group runs. Its collection
1007
+ runs without it, and named on its own runs every step:
1008
+
1009
+ ```yaml
1010
+ id: orders
1011
+ steps:
1012
+ - name: create order
1013
+ POST: '{{baseUrl}}/orders'
1014
+ - name: refund
1015
+ POST: '{{baseUrl}}/orders/{{orderId}}/refund'
1016
+ exclude: true # waiting on a fix; ata orders still runs it
1017
+ ```
1018
+
1019
+ - It is only on a step of `steps`. Setup and teardown run whenever their collection
1020
+ does, and a reusable requests file's steps wherever it is used, so neither has it:
1021
+ exclude the collection, or the use step, instead.
1022
+ - A step the tags would not pick is not run either way. `ata` lists each step it left
1023
+ out as `orders › refund`.
1024
+ - A collection whose every step is excluded is left out of a group run, setup and
1025
+ teardown too.
1026
+ - In the desktop app, Run all leaves it out too, and Send still sends it.
1027
+ - Leaving out a step can make later steps fail, such as one that uses a value it sets.
1028
+
1029
+ ### 2.5 Reusable requests and `use:`
1030
+
1031
+ **Reusable requests** are one or more requests that any step can run with **`use:`**,
1032
+ passing values with **`with:`**. They are written in a **reusable requests file**: a
1033
+ collection in `requests/`, directly or one folder down, with a `params:` key, the
1034
+ inputs it takes.
1035
+
1036
+ ```yaml
1037
+ # requests/login.yml
1038
+ id: login
1039
+ params:
1040
+ username: { required: true, description: Account to log in as }
1041
+ password: { required: true }
1042
+ expectStatus: 200 # a default
1043
+ steps:
1044
+ - name: log in
1045
+ POST: '{{baseUrl}}/login'
1046
+ body:
1047
+ json: '{ "user": "{{params.username}}", "password": "{{params.password}}" }'
1048
+ tests: |
1049
+ ata.expectResponseStatusCodeToBe(params.expectStatus)
1050
+ if (params.expectStatus === 200) {
1051
+ ata.expectResponseBodyToHaveProperty('token', 'authToken', 'setAsCollectionVariable')
1052
+ }
1053
+ ```
1054
+
1055
+ ```yaml
1056
+ # collections/checkout.yml
1057
+ id: checkout
1058
+ steps:
1059
+ - use: login
1060
+ with:
1061
+ username: '{{adminUser}}'
1062
+ password: '{{adminPassword}}'
1063
+ - use: login
1064
+ name: rejects a bad password
1065
+ with: { username: alice, password: wrong, expectStatus: 401 }
1066
+ tests: |
1067
+ ata.expectResponseBodyToHaveProperty('error', 'invalid credentials')
1068
+ - name: get profile
1069
+ GET: '{{baseUrl}}/me'
1070
+ ```
1071
+
1072
+ **Params.** Each param is a plain default (`expectStatus: 200`), or
1073
+ `{ required: true, default, description }`. In a request a param reads as
1074
+ `{{params.name}}`, and in code as `params.name`, a read-only object holding the real
1075
+ values. No variable can take the place of a `params.` name.
1076
+
1077
+ A default may name variables and other params, as in
1078
+ `email: '{{params.accountId}}@example.com'`. Like a `with:` value, it is resolved once
1079
+ for each use, so `accountId: '{{$uuid}}'` is one id wherever the file reads it.
1080
+
1081
+ **A use step** holds only `use`, `with`, `name`, `tags`, `flags`, `docs` and `tests`. A
1082
+ method key, `headers`, `body`, `settings` or `before` on it is an error, and `with`
1083
+ without `use` is an error too.
1084
+
1085
+ - **Finding the file.** `use: login` is `requests/login.yml` in the project, else in its
1086
+ global project. `use: auth/login` is one folder down. `use: global:login` looks
1087
+ only in the global project.
1088
+ - **`with:`** gives plain values; a param left out takes its default. A string may hold
1089
+ `{{variables}}`, resolved as the file's first request starts, just after the
1090
+ collection's `before.script` has run for it: a value that script sets for each step
1091
+ (§4) reaches the file's requests. A missing required value, a name the file does not
1092
+ take, or a value or default that cannot be resolved stops the file's steps before
1093
+ anything is sent.
1094
+ - **Running.** A use step runs each of the file's steps in turn, in the collection's
1095
+ variable scope, so what one sets the next can read, and so can the steps after the use
1096
+ step. Each request is reported as its own result. When the use step has a `name`,
1097
+ reports use it: `sign in` for a file of one step, `sign in › get profile` for a longer
1098
+ one.
1099
+ - **Layers.** Headers and settings: the collection's, under the file's, under each
1100
+ step's own. Scripts run collection, then file, then step: `before.script` before the
1101
+ request and `tests` after. The use step's own `tests` run last, on the response of
1102
+ the file's step marked `useTests: true`, or else its last step that sends a request:
1103
+ a wait has no response. A use step with `tests` cannot run a file of only waits.
1104
+ `params` is the file's alone: the collection's scripts, and a base collection's or an
1105
+ endpoint's (§2.6, §2.7), never see it. In an endpoint's `before.script`, a segment written as
1106
+ `{{params.id}}` reads as written.
1107
+ - **One level.** A reusable requests file must not `use:` another, and has `params`,
1108
+ not `vars`. A file in `requests/` without `params` is not a reusable requests file.
1109
+
1110
+ **Saving under the caller's name.** A reusable requests file can take the name to save
1111
+ a value under as a param, save it with `ata.set(params.saveAs, …)`, and read it back in
1112
+ its later steps with `{{@params.saveAs}}` (§4). The caller then reads it by the name it
1113
+ chose:
1114
+
1115
+ ```yaml
1116
+ # requests/create-user.yml
1117
+ params:
1118
+ saveTokenAs: { required: true }
1119
+ steps:
1120
+ - name: create token
1121
+ POST: '{{authUrl}}/token'
1122
+ tests: ata.set(params.saveTokenAs, res.body.accessToken)
1123
+ - name: get profile
1124
+ GET: '{{baseUrl}}/profile'
1125
+ headers:
1126
+ Authorization: Bearer {{@params.saveTokenAs}}
1127
+ useTests: true # the use step's tests check this response
1128
+ - name: wait for events
1129
+ GET: '{{baseUrl}}/wait'
1130
+ ```
1131
+
1132
+ ```yaml
1133
+ # a collection
1134
+ - use: create-user
1135
+ name: user 1
1136
+ with: { saveTokenAs: token1 }
1137
+ tests: ata.expectResponseBodyToHaveProperty('email') # on get profile's response
1138
+ - GET: '{{baseUrl}}/orders'
1139
+ headers: { Authorization: 'Bearer {{token1}}' }
1140
+ ```
1141
+
1142
+ Only one step of a reusable requests file may have `useTests`, and no other file's
1143
+ steps.
1144
+
1145
+ ### 2.6 Endpoint bases
1146
+
1147
+ What every request to one endpoint gets, wherever the request is written. A file in
1148
+ `endpoints/` (the project's own, or its global project's) is a collection whose steps
1149
+ are **endpoints**: a method and a **path pattern**, with the headers, settings, scripts
1150
+ and checks for every request to it.
1151
+
1152
+ ```yaml
1153
+ # endpoints/users.yml
1154
+ id: users
1155
+ headers:
1156
+ X-Api: users # every endpoint in this file
1157
+ steps:
1158
+ - GET: /users/{id}
1159
+ headers:
1160
+ Accept: application/json
1161
+ tests: |
1162
+ ata.expectResponseStatusCodeToBe(200)
1163
+ ata.expectResponseBodyToHaveProperty('id', endpoint.id)
1164
+ - POST: /users
1165
+ tests: |
1166
+ ata.expectResponseStatusCodeToBe(201)
1167
+ ```
1168
+
1169
+ - An endpoint's URL must be a path starting with `/`. An endpoints file must not hold a
1170
+ use step or a wait step.
1171
+ - **Matching.** A step's request is under the endpoint with its method and a matching
1172
+ path. The path is read from the URL as written. The host, or a leading
1173
+ `{{variable}}` standing for it, is ignored, and so is the query string, so one base
1174
+ serves every environment.
1175
+ - `{name}` in a pattern stands for one path segment, whether a literal value or a
1176
+ `{{variable}}`. A variable never matches a literal segment of a pattern.
1177
+ - When several endpoints match, the one with the most literal segments wins
1178
+ (`/users/me` over `/users/{id}`), then the one listed first. A project's endpoint
1179
+ replaces its global project's with the same method and path.
1180
+ - **`endpoint`** in scripts holds each `{name}`'s value from the step's URL, resolved:
1181
+ `endpoint.id` is `42` for `{{baseUrl}}/users/{{userId}}` with `userId: 42`.
1182
+ - **What applies**, outermost first: the endpoints file's own `headers`, `settings`,
1183
+ `before` and `tests`, then the endpoint's, then the base collection's (§2.7), the
1184
+ collection's, the reusable requests file's (§2.5) and the step's. Nearer headers and
1185
+ settings win.
1186
+ - **A step's own check replaces the base's check of the same thing**: the status, a
1187
+ header by name, or a body property by path. A negative test only says what it expects,
1188
+ so checking for a 404 replaces the base's 200. Checks of other things stay, and named
1189
+ tests (`ata.test`) are never replaced.
1190
+ - **`base: false`** on a step leaves its endpoint base out altogether.
1191
+
1192
+ ### 2.7 `extends`: base collections
1193
+
1194
+ ```yaml
1195
+ # bases/authenticated.yml
1196
+ id: authenticated
1197
+ headers:
1198
+ Authorization: Bearer {{token}}
1199
+ before:
1200
+ script: |
1201
+ ata.set('requestId', ata.uuidv7())
1202
+ ```
1203
+
1204
+ ```yaml
1205
+ # collections/checkout.yml
1206
+ id: checkout
1207
+ extends: authenticated
1208
+ steps:
1209
+ - GET: '{{baseUrl}}/cart'
1210
+ ```
1211
+
1212
+ A collection that `extends:` a base collection builds on its `headers`, `settings`,
1213
+ `vars`, `before` and `tests`, with its own on top: nearer headers, settings and
1214
+ variables win, and scripts run the base's first.
1215
+
1216
+ - `extends: name` looks in the project's `bases/`, then its global project's.
1217
+ `extends: global:name` looks only in the global project's.
1218
+ - A base collection must have no `steps` and no `params`, and must not `extends`
1219
+ another.
1220
+ - A base that cannot be used (missing, broken, or breaking these rules) stops every
1221
+ step of the collection before anything is sent, saying why.
1222
+
1223
+ ### 2.8 Data files
1224
+
1225
+ A collection can be driven by a **data file**: `<id>.csv` or `<id>.json` beside
1226
+ `<id>.yml`, with exactly the same name. The whole collection runs once per row, and each
1227
+ column of the row is a variable for that run: `{{userId}}`, or `ata.get('userId')`.
1228
+
1229
+ ```
1230
+ collections/
1231
+ ├── users.yml
1232
+ └── users.csv
1233
+ ```
1234
+
1235
+ ```csv
1236
+ userId,expectedStatus,iterationLabel
1237
+ 1001,200,Happy path
1238
+ 9999,404,Unknown user
1239
+ ```
1240
+
1241
+ - **One run per row.** Every step runs for row 1, then every step again for row 2, and
1242
+ so on. Each run starts from a fresh variable scope, so nothing one row sets leaks into
1243
+ the next, except a value set to last the run (§2.10). Three steps and two rows report
1244
+ six results.
1245
+ - **Precedence.** A row's values sit over the environment and under what code sets
1246
+ (§4).
1247
+ - **`iterationLabel`**, an optional column, names the row in reports, as in
1248
+ `Iteration 2 (Unknown user) - get user`. It is a variable like the others.
1249
+ - **CSV** follows RFC 4180: a header row, then one row per line. A value in double
1250
+ quotes may hold commas and line breaks, and `""` is a quote. **Every CSV value is a
1251
+ string**: a zip code `01234` stays `01234`. A short row leaves its last columns empty;
1252
+ a row with more values than the header is an error.
1253
+ - **JSON** is an array of objects, one per row. Values keep their type, which must be
1254
+ string, number, boolean or null. A row must not name a column twice: JSON readers keep
1255
+ only the last value, so it is an error rather than a value quietly lost.
1256
+ - **`.csv` wins** when both exist. The name must match exactly, case included.
1257
+ - A data file must be at most 10 MB and have at least one row. The header must not have
1258
+ an empty or repeated column name. A data file that will not read makes the collection
1259
+ broken: `ata` reports it and runs nothing from it.
1260
+ - With `bail`, a failing step also stops the rows still to come.
1261
+
1262
+ In the desktop app, a single step runs with one chosen row. **Run all** runs every row,
1263
+ as `ata` does.
1264
+
1265
+ ### 2.9 Feature flags
1266
+
1267
+ A collection or a step says which feature flags it needs, and runs only when they hold:
1268
+
1269
+ ```yaml
1270
+ id: checkout
1271
+ flags: { newCheckout: true } # the whole collection runs only when newCheckout is on
1272
+ steps:
1273
+ - name: total (new)
1274
+ GET: '{{baseUrl}}/checkout/total'
1275
+ flags: { newCheckout: true }
1276
+ - name: total (old)
1277
+ GET: '{{baseUrl}}/checkout/total'
1278
+ flags: { newCheckout: false } # only one of the pair ever runs
1279
+ - name: v2 pricing
1280
+ GET: '{{baseUrl}}/prices'
1281
+ flags: { pricingVersion: v2 } # any value, not just on/off
1282
+ ```
1283
+
1284
+ - **Every flag named must have the value given.** A collection's `flags` apply to each
1285
+ of its steps, and a use step's to each of its reusable requests. Values compare as
1286
+ text, so `true` matches `true` or `"true"`, and `2` matches `"2"`.
1287
+ - **A step whose flags do not hold is skipped**, not failed. It sends nothing and is
1288
+ reported as skipped with the reason, such as `feature flag newCheckout is off`. A
1289
+ skipped step never fails a run.
1290
+ - **A flag the run does not know is an error.** Otherwise a typo, or a flag deleted from
1291
+ the flag service, would quietly run or skip the wrong tests.
1292
+ - **In code, `ata.flag(name)`** returns a flag's value, for checking something
1293
+ different rather than skipping a step.
1294
+ - A flag name is letters, digits and `- _ .` (`^[A-Za-z0-9_][A-Za-z0-9_.-]*$`). A value
1295
+ is a string, number or boolean.
1296
+ - Skipping an early step can make later steps fail, such as a login that sets a token.
1297
+ Flags on the whole collection are usually the safer choice.
1298
+
1299
+ **Where the values come from.** Each environment file gives its flags (§6):
1300
+
1301
+ ```yaml
1302
+ # environments/staging.yml
1303
+ vars: { baseUrl: https://staging.example.com }
1304
+ flags:
1305
+ command: node scripts/flags.mjs staging # prints the flags as JSON
1306
+ values: # used without a command, and for any flag its output leaves out
1307
+ newCheckout: false
1308
+ ```
1309
+
1310
+ Lowest precedence first: the global project's environment's `values`, the project's
1311
+ own, what the `command` prints, then overrides.
1312
+
1313
+ - **`command`** runs a program once before any test of a run, in the folder of the
1314
+ project whose environment file names it. It must print, on standard output, a JSON
1315
+ object of flag names to string, number or boolean values, such as
1316
+ `{ "newCheckout": true, "pricingVersion": "v2" }`.
1317
+ - **It runs the same way on every platform, without a shell.** The command is split into
1318
+ words at spaces. The first word is the program, found on the `PATH` or given as a path
1319
+ from the project folder, and the rest are its arguments. `'…'` or `"…"` keeps spaces
1320
+ in a word. Inside `"…"`, `\"` stands for `"` and `\\` for `\`; anywhere else `\` is an
1321
+ ordinary character, so a Windows path works as written. Pipes, `&&`, redirection,
1322
+ `$VAR` and `%VAR%` mean nothing. Put logic in a script and run it with its interpreter,
1323
+ as in `node scripts/flags.mjs staging`. On Windows, a `.cmd` or `.bat` script such as
1324
+ `npx` cannot be started this way, so name the program it runs instead.
1325
+ - It inherits the environment `ata` or the app runs in, so a flag service's API key
1326
+ comes from CI or the shell, never from the repository. The desktop app adds the `PATH`
1327
+ a terminal would have, so `node` is found however the app was opened.
1328
+ - If it exits with anything but `0`, prints anything else, or takes longer than 60
1329
+ seconds, **nothing runs**, and what it wrote to standard error is shown. A command that
1330
+ takes too long is stopped, with anything it started.
1331
+ - A project's command wins over its global project's for the same environment.
1332
+ - **Overrides** win over everything: `ata … --flag newCheckout=false` (any number of
1333
+ them), or the environment variable `ATA_FLAG_newCheckout=false`. The desktop app has an
1334
+ override per flag. `true` and `false` are booleans, a number is a number, and anything
1335
+ else is text.
1336
+
1337
+ Every report records the flag values a run used and where each came from.
1338
+
1339
+ ### 2.10 Setup, teardown and values that last the run
1340
+
1341
+ `setup` and `teardown` are lists of steps, like `steps`, run once around it:
1342
+
1343
+ ```yaml
1344
+ id: approved-domains
1345
+ setup: # once, before the first row
1346
+ - use: seed-admin # log in, grant an admin role, wait for it to land
1347
+ teardown: # once, after the last row, even when a row failed
1348
+ - name: remove the grant
1349
+ DELETE: '{{baseUrl}}/admins/{{adminId}}'
1350
+ steps: # once per row of approved-domains.csv, as before
1351
+ - name: approve the domains
1352
+ PUT: '{{baseUrl}}/orgs/{{orgId}}/domains'
1353
+ ```
1354
+
1355
+ - **Order.** Setup, then the steps once per data row (or once, with no data file), then
1356
+ teardown. They work the same with or without a data file.
1357
+ - **What setup sets lasts the run.** Every value setup sets or captures is there for
1358
+ every row and for teardown. It sits over the environment and under a row's values
1359
+ (§4).
1360
+ - **A row keeps a value for the rows after it** with
1361
+ `ata.set(name, value, { scope: 'run' })`. Anything else a row sets lasts only that
1362
+ row, as §2.8 says.
1363
+ - **Setup stops at its first step that fails**, with `bail` or without: a setup step
1364
+ counts on the ones before it. The setup steps after it and the rows are not run, and
1365
+ count as skipped. Teardown still runs.
1366
+ - **Teardown always runs**, every step of it, after failures and after `bail` — and after
1367
+ a run is cancelled or stopped by `timeoutCollection`, when it has at most 60 seconds of
1368
+ its own. A cancel or timeout before anything ran runs no teardown, since there is
1369
+ nothing to clean up after. One that comes while teardown runs stops it where it is,
1370
+ and so does a second Stop in the desktop app.
1371
+ - **The collection applies to them**: its headers, settings, `before`, `tests` and
1372
+ flags, as to any step. Their steps may be use steps, and only theirs may have
1373
+ `forEach` (§2.1). They carry no `tags`: they run whenever the collection does.
1374
+ - A reusable requests file, a base collection and an endpoints file have no setup or
1375
+ teardown.
1376
+ - Reports name their results `setup › log in` and `teardown › remove the grant`. Running
1377
+ a single step in the desktop app does not run them.
1378
+
1379
+ ### 2.11 Connections: a stream across steps
1380
+
1381
+ A request whose response is an event stream (§2.3) can keep it open for later steps, as
1382
+ a **connection** named by `connection:`. A later step with `connection:` and no method
1383
+ key sends nothing: it reads the events the connection holds. The steps between can do
1384
+ what those events are about:
1385
+
1386
+ ```yaml
1387
+ steps:
1388
+ - name: watch orders
1389
+ GET: '{{baseUrl}}/orders/events'
1390
+ connection: orders # keep the stream open as "orders"
1391
+ settings:
1392
+ untilEvent: subscribed # read until the server confirms, then go on
1393
+ - name: place order
1394
+ POST: '{{baseUrl}}/orders'
1395
+ body:
1396
+ json: '{ "sku": "ACME-1" }'
1397
+ tests: |
1398
+ ata.expectResponseBodyToHaveProperty('id', 'orderId', 'setAsCollectionVariable')
1399
+ - name: order created
1400
+ connection: orders # no method key: reads the connection
1401
+ settings:
1402
+ streamTimeout: 5000 # read until the tests pass, for up to 5 s
1403
+ tests: |
1404
+ ata.test('the order placed', () => {
1405
+ const last = res.body.at(-1)
1406
+ assert.equal(last.event, 'order.created')
1407
+ assert.equal(last.data.id, ata.get('orderId'))
1408
+ })
1409
+ ```
1410
+
1411
+ - **Opening.** The step sends its request and reads the stream as any step does, until
1412
+ its `maxEvents`, `streamTimeout` or `untilEvent`, or its tests pass. Then, rather than
1413
+ closing it, it keeps reading in the background, holding the events that arrive for
1414
+ the next step that reads the connection. With none of the three set, it reads no
1415
+ events and goes straight on: `untilTestsPass` alone does not make it wait.
1416
+ - **Reading.** A reading step takes the events held since the last step read them,
1417
+ then waits for more, until its `maxEvents`, `streamTimeout` or `untilEvent`, its tests
1418
+ pass, or the server closes the stream. With none of the three set, it takes what is
1419
+ held and ends without waiting, as `held`, or sooner, as `testsPassed`, at the event
1420
+ its tests pass on.
1421
+ - **What a step leaves.** The events after the one a step stopped at stay held for the
1422
+ next step that reads the connection.
1423
+ - **What a reading step sees.** Its body is the events it took (§3). Its status and
1424
+ headers are those of the response that opened the connection, `req` is that request,
1425
+ and `res.stream.at` counts from that response's headers.
1426
+ - A reading step may have `name`, `settings`, `before`, `tests`, `tags`, `flags`,
1427
+ `useTests` and `docs`, and nothing that would build a request: no `headers`, `body`,
1428
+ `base` or `forEach`. A step opening a connection has no `forEach`, and a use step has
1429
+ no `connection`.
1430
+ - **A connection lasts the run.** One opened in `setup` is read by every data row; one
1431
+ opened in a row, by that row's later steps. Every connection closes when the run ends,
1432
+ cancelled or not. A step opening a name already open closes the old connection first,
1433
+ whatever its own response is.
1434
+ - A connection the server closes keeps the events it held: the next step reads them,
1435
+ then ends with `close`. A connection holds at most 1,000 events or 10 MB; at that it
1436
+ stops reading, and the step that reads it ends with `limit`.
1437
+ - Reading a connection no step has opened is an error, and the step's scripts do not
1438
+ run.
1439
+ - **In the desktop app**, a connection opened by sending a step stays open for the steps
1440
+ sent after it, until that step is sent again, the connection is closed from above the
1441
+ step list, Run all starts, or the app quits. Run all, like `ata`, opens its own and
1442
+ closes them when it ends.
1443
+
1444
+ ---
1445
+
1446
+ ## 3. Checking a response
1447
+
1448
+ A step's checks are calls to functions on `ata`, in its `tests` (§5 lists them).
1449
+ [FUNCTIONS.md](./FUNCTIONS.md) documents each one, with examples. This section is the
1450
+ rules they share.
1451
+
1452
+ ```yaml
1453
+ tests: |
1454
+ ata.useStrictValidation()
1455
+ ata.expectResponseStatusCodeToBe(200)
1456
+ ata.expectResponseToHaveHeader('Content-Type', /^application\/json/)
1457
+ ata.expectResponseBodyToHaveProperty('user.name', 'Ada')
1458
+ ata.expectResponseBodyToHaveProperty('user.score', 100, 'integerWithin2')
1459
+ ata.expectResponseBodyToHaveProperty('user.token', 'sessionToken', 'setAsCollectionVariable')
1460
+ ata.expectResponseBodyToHaveUnorderedArray('user.roles', ['admin', 'editor'])
1461
+ ata.ignoreResponseBodyProperty('user.lastSeen')
1462
+ ```
1463
+
1464
+ ### Paths
1465
+
1466
+ A path addresses the body in dot or bracket notation: `user.name`; `groups.0.name` or
1467
+ `groups[0].name` for an index; and `sessions[].id` for a property of every item.
1468
+
1469
+ A key that holds a `.`, `[` or `]`, or is empty, goes in brackets as a JSON string:
1470
+ `jwt.payload["https://example.com/id"]`, `modules[""].edition`. Reports show such keys
1471
+ the same way.
1472
+
1473
+ A path can also be a list of keys: `['jwt', 'payload', 'https://example.com/id']`.
1474
+ Each item is one key, whatever it holds, and a number is its digits, so
1475
+ `['groups', 0, 'name']` reads an index. Every function that takes a path takes a list
1476
+ too, and so does `pathToProperty`.
1477
+
1478
+ A path that runs into a `null` before its end, such as `phoneNumber.number` when
1479
+ `phoneNumber` is `null`, reads as that `null` for a check that the value is `null`. For
1480
+ any other check the property is not present.
1481
+
1482
+ ### Body conversion
1483
+
1484
+ A JSON body is used as it is. An XML body is converted: the root element is the single
1485
+ top-level key, namespace prefixes and attributes are dropped, an element holding only
1486
+ text becomes that string (an empty one `""`), and a repeated element becomes an array.
1487
+ A `text/plain` or HTML body is the single property `plaintext`.
1488
+
1489
+ An event stream (§2.3) is a list with one item per event:
1490
+
1491
+ - **`data`** is the event's `data:` lines joined with a line feed: their JSON value when
1492
+ they parse as JSON, so `[1].data.price` is a number, and otherwise the text, such as
1493
+ `[DONE]`.
1494
+ - **`event`** and **`id`** are there only when that event's own lines set them. Neither
1495
+ carries over to the next event.
1496
+ - Comments (`: heartbeat`) and a block with no `data:` are not events, `retry:` is not
1497
+ kept, and an event the stream ended in the middle of is dropped.
1498
+
1499
+ ```text
1500
+ event: price
1501
+ id: 41
1502
+ data: {"symbol":"ACME","price":100}
1503
+
1504
+ : heartbeat
1505
+
1506
+ data: [DONE]
1507
+
1508
+ ```
1509
+
1510
+ reads as
1511
+
1512
+ ```json
1513
+ [{ "event": "price", "id": "41", "data": { "symbol": "ACME", "price": 100 } }, { "data": "[DONE]" }]
1514
+ ```
1515
+
1516
+ The path `''` is the whole list, so `expectResponseBodyToHaveUnorderedArray('', …)` finds
1517
+ events whose order is not guaranteed. Under strict validation every event's properties
1518
+ need accounting for, so a small `maxEvents` keeps a busy stream practical.
1519
+
1520
+ ### Values and patterns
1521
+
1522
+ - A value compares with its type: `'12345'` does not equal `12345`.
1523
+ - A `RegExp` is a pattern, flags included, tested against the value as text.
1524
+ - Header names match case-insensitively. The status and header values compare as text.
1525
+
1526
+ ### `specialHandling`
1527
+
1528
+ The last argument of a check may be one of these strings:
1529
+
1530
+ | String | Means |
1531
+ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
1532
+ | _(none, and no value)_ | The property or header exists. |
1533
+ | `notThisExpectedKey` | It must not exist. Pass `null` as the value. |
1534
+ | `notThisExpectedValue` | It must exist and not equal the value (or not match the RegExp). |
1535
+ | `setAsCollectionVariable` | Capture it into the variable the value names, for the rest of the run. |
1536
+ | `setAsEnvironmentVariable` | The same. Nothing is written back to the environment file. |
1537
+ | `dateAsEpoch` | Compare by calendar day: a number is a seconds offset from when the step started, a string a date whose first ten characters must match. |
1538
+ | `dateWithin<X>Sec` | A date within X seconds of the value, as in `dateWithin5Sec`. |
1539
+ | `integerWithin<X>` | A number within X of the value, as in `integerWithin2`. |
1540
+ | `isArray` | An array; its contents are not checked. |
1541
+ | `isArrayAndEmpty` / `isArrayAndNotEmpty` | An empty or non-empty array. |
1542
+ | `isArrayAndHasLength` | An array of exactly the value's length. |
1543
+
1544
+ A mistake in a call, such as an unknown `specialHandling`, is reported as a failed check
1545
+ saying so, and the checks after it still run.
1546
+
1547
+ ### Unordered arrays
1548
+
1549
+ `ata.expectResponseBodyToHaveUnorderedArray(path, list)` passes when the array holds
1550
+ every item of a simple `list`, in any order. A `RegExp` in the list is a pattern some item
1551
+ must match as text, and so is one held by an object in the list:
1552
+ `[/^admin/, { name: /^Grace/ }]`. A pattern never matches an object or array item.
1553
+ An object in the list matches an item holding each of its properties, and the item may
1554
+ have others; a property that is itself an object is matched the same way, so
1555
+ `{ data: { status: 'reversed' } }` finds an item whose `data` has that `status` among
1556
+ other properties. An array compares whole.
1557
+
1558
+ A list of `{ pathToProperty, expectedValue, specialHandling? }` objects describes **one**
1559
+ item, property by property; call it once per item. A property may appear twice, once to
1560
+ check it and once to capture it:
1561
+
1562
+ ```js
1563
+ ata.expectResponseBodyToHaveUnorderedArray('users', [
1564
+ { pathToProperty: 'name', expectedValue: 'Ada' },
1565
+ { pathToProperty: 'id', expectedValue: 'adaId', specialHandling: 'setAsCollectionVariable' }
1566
+ ])
1567
+ ```
1568
+
1569
+ `ata.expectResponseBodyToHaveUnorderedArrayNotThisItem(path, list)` passes when no item
1570
+ matches. A simple list may hold patterns here too, and a `compareValue` may be a
1571
+ `RegExp`.
1572
+
1573
+ - **Each call prefers items an earlier call did not match.** Two calls with the same
1574
+ description find two items when there are two, so each capture and strict validation
1575
+ see a different one. A sort starts this over, since its indexes name other items.
1576
+ - **A list of one `notThisExpectedValue` entry depends on strict validation.** Without
1577
+ it, the entry means no item has that value, so an empty array passes. With it, the
1578
+ entry means one item whose value is something else, as any list does. The step's last
1579
+ word on strict validation decides.
1580
+
1581
+ ### Strict validation
1582
+
1583
+ `ata.useStrictValidation()` fails the step unless **every** property of the body is
1584
+ checked, ignored or captured.
1585
+
1586
+ - `null`, `""` and empty arrays or objects never need a check of their own.
1587
+ - A check on a value's content accounts for that value and everything beneath it. A
1588
+ shape-only check (`isArray`, `isArrayAndHasLength`, or an object's existence) does not
1589
+ vouch for what is inside.
1590
+ - It is judged after the collection's and the step's `tests` have both run, over what
1591
+ either checked.
1592
+ - A binary or HTML body has no properties to leave unchecked, so strict validation
1593
+ passes. A check on one still fails.
1594
+
1595
+ ### Sorting
1596
+
1597
+ `ata.sortResponseBodyArrays(property)` sorts every array of objects holding the
1598
+ property, before the checks after it.
1599
+
1600
+ - The property may be a path (`id.value`), and nested arrays are sorted too.
1601
+ - Items without the property go last.
1602
+ - Values compare alphanumerically, so `Group 2` comes before `Group 10`.
1603
+ - Indexed paths refer to the sorted order.
1604
+
1605
+ ---
1606
+
1607
+ ## 4. Variables
1608
+
1609
+ Variables come from `vars` in `project.yml` and in collections, the chosen environment,
1610
+ a data file's row, and what code sets while a step runs.
1611
+
1612
+ ```yaml
1613
+ vars:
1614
+ apiVersion: '2' # plain values only: string, number, boolean or null
1615
+
1616
+ before:
1617
+ script: |
1618
+ ata.set('traceId', ata.uuidv7())
1619
+ ata.set('today', ata.date('%Y-%m-%d', 0, 'utc'))
1620
+ ```
1621
+
1622
+ **`vars` hold plain data.** A value must be a string, number, boolean or null. Anything
1623
+ computed, such as an id, a date, a random number or a value built from other
1624
+ variables, is set in `before.script` with `ata.set` (§5). A collection's
1625
+ `before.script` runs before every step, so a value set there is fresh for each.
1626
+
1627
+ ### Interpolation
1628
+
1629
+ `{{name}}` resolves in the URL, headers and body, and recursively in what it resolves
1630
+ to. Whitespace inside the braces is ignored. In code, read a variable with
1631
+ `ata.get(name)` instead.
1632
+
1633
+ - **Values keep their type.** A string that is exactly one reference returns the
1634
+ variable's own value, so a variable written as `true` arrives as a boolean. Anything
1635
+ else is text, and `null` becomes the empty string inside it, except in a `json` body,
1636
+ where it is written `null`, so `"website": {{site}}` stays valid JSON.
1637
+ - **An unknown variable is an error**, never literal text. The step fails in its own
1638
+ `interpolate` phase, naming the variable, and nothing is sent. A reference loop, or a
1639
+ chain more than 16 deep, fails the same way.
1640
+ - **`{{@name}}` reads the variable `name` names.** With `saveAs: token1`,
1641
+ `{{@saveAs}}` is the value of `token1`. The name may itself be built from variables.
1642
+ A name that is not text, or names no variable, fails like an unknown variable. A
1643
+ reusable requests file uses it to read what it saved under its caller's name (§2.5).
1644
+ - **In YAML, quote a value that starts with `{{`.** Unquoted, YAML reads `{` as a map.
1645
+
1646
+ ### Built-in variables
1647
+
1648
+ These are usable anywhere a variable is, with no declaration:
1649
+
1650
+ | Reference | Value |
1651
+ | ------------------- | -------------------------------------- |
1652
+ | `{{$uuid}}` | A random UUID, new for each reference. |
1653
+ | `{{$timestamp}}` | Epoch milliseconds. |
1654
+ | `{{$isoTimestamp}}` | An ISO-8601 instant. |
1655
+ | `{{$randomInt}}` | A whole number from 0 to 999. |
1656
+
1657
+ ### Resolution order
1658
+
1659
+ Lowest precedence first:
1660
+
1661
+ ```
1662
+ global project vars → project vars → base collection vars → collection vars
1663
+ → environment → what lasts the run → data file row → ata.set and captures
1664
+ → process environment
1665
+ ```
1666
+
1667
+ The environment is the global project's file of that name, if there is one, with the
1668
+ project's own over it.
1669
+
1670
+ **The process environment only overrides a name that another layer already declares.**
1671
+ Otherwise `PATH`, `HOME` and every credential on the machine would be reachable from a
1672
+ request URL. To let CI override a value, declare the variable, usually in an environment
1673
+ file and often as a secret (§6).
1674
+
1675
+ A name is matched exactly, case included, on every platform. Windows ignores case in its
1676
+ environment, but a variable called `path` or `username` is still not replaced by the
1677
+ system's `Path` or `USERNAME`.
1678
+
1679
+ ---
1680
+
1681
+ ## 5. Code: `tests` and `before.script`
1682
+
1683
+ A step, or a collection for every step, carries JavaScript. `before.script` prepares the
1684
+ request, and `tests` checks the response:
1685
+
1686
+ ```yaml
1687
+ - name: get user
1688
+ GET: '{{baseUrl}}/users/7'
1689
+ before:
1690
+ script: |
1691
+ ata.set('traceId', ata.uuidv7())
1692
+ tests: |
1693
+ ata.expectResponseStatusCodeToBe(200)
1694
+ ata.expectResponseBodyToHaveProperty('user.name', 'Ada')
1695
+ ata.expectResponseBodyToHaveProperty('user.nickname', null, 'notThisExpectedKey')
1696
+
1697
+ const ids = res.body.user.accounts.map((a) => a.id)
1698
+ ata.test('account ids are unique', () => assert.equal(new Set(ids).size, ids.length))
1699
+ ```
1700
+
1701
+ - `before` must hold only `script`.
1702
+ - Write code as a block scalar (`|`) so it needs no quoting.
1703
+ - **Order:** the collection's `before.script`, the step's `before.script`, the request,
1704
+ then the collection's `tests` and the step's `tests`. Strict validation is judged last.
1705
+
1706
+ ### The `ata` object
1707
+
1708
+ There is nothing to import or load. [FUNCTIONS.md](./FUNCTIONS.md) documents each
1709
+ function, with examples.
1710
+
1711
+ | Function | In `before.script` |
1712
+ | ------------------------------------------------------------------------------------------------ | :----------------: |
1713
+ | `ata.expectResponseStatusCodeToBe(expected, specialHandling?)` | |
1714
+ | `ata.expectResponseToHaveHeader(name, expected?, specialHandling?)` | |
1715
+ | `ata.expectResponseToHaveHeaderCount(name, expectedCount)` | |
1716
+ | `ata.expectResponseBodyToHaveProperty(path, expected?, specialHandling?)` | |
1717
+ | `ata.expectResponseBodyToHaveUnorderedArray(path, list)` | |
1718
+ | `ata.expectResponseBodyToHaveUnorderedArrayNotThisItem(path, list)` | |
1719
+ | `ata.ignoreResponseBodyProperty(path)` | |
1720
+ | `ata.ignoreResponseBodyArrayObjectProperty(arrayPath, propertyPath)` | |
1721
+ | `ata.sortResponseBodyArrays(property)` | |
1722
+ | `ata.useStrictValidation(enabled = true)` | |
1723
+ | `ata.test(name, fn)`: a named check that passes unless `fn` throws or rejects; `fn` may be async | |
1724
+ | `ata.get(name)`: a variable's current value | ✓ |
1725
+ | `ata.set(name, value)`: set a variable for the rest of the run | ✓ |
1726
+ | `ata.set(name, value, { scope: 'run' })`: the same, lasting past this data row too (§2.10) | ✓ |
1727
+ | `ata.skip(reason?)`: send nothing for this step, and report it skipped with the reason | ✓ |
1728
+ | `ata.skipRest(reason?)`: skip the rest of this row's steps; in `before.script`, this one too | ✓ |
1729
+ | `ata.flag(name)`: a feature flag's value; an unknown flag is an error | ✓ |
1730
+ | `ata.uuid()`: a random (version 4) UUID | ✓ |
1731
+ | `ata.uuidv7()`: a time-ordered (version 7) UUID | ✓ |
1732
+ | `ata.randomInt(min, max)`: a whole number, both ends included | ✓ |
1733
+ | `ata.date(format, secondsOffset = 0, timeZone = 'local')` | ✓ |
1734
+
1735
+ `ata.set` keeps a string, number, boolean or null as it is, and stores an object or array
1736
+ as JSON text. `undefined` is stored as `null`, so a variable set to nothing still reads
1737
+ `== null`, and `{{name}}` resolves as a null does (§4).
1738
+
1739
+ Calling a response check in `before.script` is an error, and so is `ata.skip` in
1740
+ `tests`, where the request has been sent. After `ata.skip` the script runs to its end,
1741
+ and no later `before.script` runs. `ata.skipRest` skips steps of this row only: the next
1742
+ row, and teardown, still run.
1743
+
1744
+ `ata.date` formats with strftime specifiers: `%Y %y %m %d %e %H %I %M %S %L %p %b %B %a
1745
+ %A %j %Z %z %s %F %T %%`. An unrecognized specifier is left in the output, so a typo is
1746
+ visible. `timeZone` is `local`, `utc`, an IANA name such as `America/New_York`, or a
1747
+ military zone letter (`U` is -08:00, not UTC).
1748
+
1749
+ ### Other globals
1750
+
1751
+ | Global | What it is |
1752
+ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
1753
+ | `res` | `tests` only: `status`, `statusText`, `headers` (lower-cased names, a repeated header's values joined with `, `), `header(name)`, `headerValues(name)` (every value, as a list), `body` (parsed JSON, converted XML, an event stream's events, or text), `text`, `time` (ms), `size` (bytes), and for an event stream `stream` (below). |
1754
+ | `req` | `method`, `url`, `headers`, `body`: as sent in `tests`, as written in `before.script`, where a script may change `headers` and `body` (below). |
1755
+ | `assert` | Node's strict `assert`, for use inside `ata.test`. |
1756
+ | `console` | Captured into the step's result. |
1757
+ | `params` | A reusable requests file's params (§2.5), in its own scripts and in the tests of the use step running it. |
1758
+ | `endpoint` | An endpoint base's `{name}` values (§2.6), in its scripts and in every script of a step under it. |
1759
+ | `checks` | The project's check files (below). |
1760
+
1761
+ **`res` for an event stream** (§2.3): `res.body` is its list of events, as checks see it
1762
+ (§3), `res.text` the stream as received, and `res.time` runs until the reading stopped.
1763
+ `res.stream` has the rest, and is absent for any other response:
1764
+
1765
+ - `endedBy`: what stopped the reading: `close`, `maxEvents`, `streamTimeout`,
1766
+ `untilEvent`, `testsPassed` (`untilTestsPass`), `limit`, `stopped` (the desktop app's
1767
+ Stop button), or `held` (a step on a connection that waited for nothing, §2.11)
1768
+ - `at`: for each event, the milliseconds from the response headers to its arrival
1769
+ - `connection`: for a step that opens or reads a connection, `{ name, open }`: its name,
1770
+ and whether it was still open when the step ended
1771
+
1772
+ ```js
1773
+ ata.test('first price within 2s', () => assert.ok(res.stream.at[0] < 2000))
1774
+ ```
1775
+
1776
+ Also available are the language itself and the web-standard globals: timers, `URL`,
1777
+ `URLSearchParams`, `TextEncoder`, `TextDecoder`, `atob`, `btoa`, `structuredClone` and
1778
+ `crypto`.
1779
+
1780
+ **There is no `require`, `import`, `process`, filesystem or `fetch`.** A collection must
1781
+ run the same on any machine, and a request belongs in a step, where it is recorded.
1782
+
1783
+ - A script that throws stops that script. It is reported with its line, and the step is
1784
+ marked as errored. Checks made before it are kept.
1785
+ - A script that runs for more than 10 seconds is stopped.
1786
+
1787
+ **Changing the request in `before.script`.** `req` there is the request as written,
1788
+ `{{variables}}` still in it. A script may change it before it is sent, and a later
1789
+ `before.script` sees the change:
1790
+
1791
+ ```js
1792
+ const claims = JSON.parse(req.body)
1793
+ if (!params.crmContactId) delete claims.crm_contact_id // leave the member out
1794
+ req.body = JSON.stringify(claims)
1795
+ req.headers['X-Trace'] = '{{traceId}}'
1796
+ delete req.headers['X-Debug']
1797
+ ```
1798
+
1799
+ - `req.body` is text, and can be changed for a `json`, `xml`, `text` or `graphql` body.
1800
+ A form, multipart or file body is built from its parts, so it cannot.
1801
+ - `req.headers` is a map of names to values: add, change or delete them. A header sent
1802
+ more than once reads as its values joined with `, `, as `res.headers` does, and is
1803
+ still sent once per value unless the script changes it. Set an array to send one
1804
+ header per value.
1805
+ - Variables in what the script writes resolve afterwards, as in the file.
1806
+ - Anything else is a pre-request error, and nothing is sent.
1807
+
1808
+ ### Check files
1809
+
1810
+ `checks/*.js` in a project, and in its global project, hold functions every script can
1811
+ call as `checks.<file>.<function>`:
1812
+
1813
+ ```js
1814
+ // checks/pagination.js
1815
+ export function expectPage({ size }) {
1816
+ ata.expectResponseStatusCodeToBe(200)
1817
+ ata.expectResponseBodyToHaveProperty('page.size', size)
1818
+ }
1819
+ ```
1820
+
1821
+ ```js
1822
+ // in a step's tests
1823
+ checks.pagination.expectPage({ size: 20 })
1824
+ ```
1825
+
1826
+ - A function uses the calling script's `ata`, `res`, `req`, `assert` and `params`, so
1827
+ what it checks is reported on the step that called it.
1828
+ - A file exports with `export function name`, `export async function name` or
1829
+ `export const name =`.
1830
+ - The file name, without `.js`, must be a JavaScript identifier
1831
+ (`^[A-Za-z_$][\w$]*$`). A file with any other name is not loaded.
1832
+ - A project's file replaces its global project's file of the same name.
1833
+
1834
+ ---
1835
+
1836
+ ## 6. `environments/<name>.yml`
1837
+
1838
+ ```yaml
1839
+ name: staging # optional: else the file name
1840
+ vars:
1841
+ baseUrl: https://staging.example.com
1842
+ strictValidation: true # a real boolean
1843
+ apiKey: { secret: true } # the value comes from the process environment or .env
1844
+ region: { value: eu, description: Where the test accounts live }
1845
+ flags: # feature flags for this environment (§2.9)
1846
+ command: node scripts/flags.mjs staging
1847
+ values: { newCheckout: true }
1848
+ ```
1849
+
1850
+ Any other top-level key is an error.
1851
+
1852
+ | Key | Type | Meaning |
1853
+ | ------- | ------ | ------------------------------------------------------------------------------- |
1854
+ | `name` | string | The environment's name. Defaults to the file name without `.yml`. |
1855
+ | `vars` | map | Each value is a plain value, or `{ value?, secret?: true, description? }`. |
1856
+ | `flags` | map | `command` (string) and `values` (flag name → string, number or boolean) (§2.9). |
1857
+
1858
+ **Secrets.** A variable written `{ secret: true }` never has its value in a file or a
1859
+ report. Its value is read from the process environment variable of **exactly the same
1860
+ name**, case included, else from `.env` at the project's root, else from `.env` at its
1861
+ global project's root.
1862
+
1863
+ - A secret with no value anywhere fails the run, naming it. An empty credential is never
1864
+ sent.
1865
+ - Reports replace a secret's value with `[secret: NAME]`, wherever it appears. So does
1866
+ the desktop app where it shows a request as sent, or what a script wrote with
1867
+ `console`.
1868
+ - `.env` holds `NAME=value` lines. `#` starts a comment line, `export ` before a name is
1869
+ allowed, and a value may be quoted. It must not be committed.
1870
+
1871
+ An environment is selected by its `name`, else its file name. A project can use its own
1872
+ `environments/` and its global project's (§1.1). Two files of the same name are one
1873
+ environment, with the project's values over the global project's.
1874
+
1875
+ ---
1876
+
1877
+ ## 7. Editing files
1878
+
1879
+ Files are meant to be written by hand, by tools and by agents, and kept in git.
1880
+
1881
+ - **Accelerator edits in place.** Saving a file Accelerator did not change leaves it as it was,
1882
+ byte for byte. Changing one field changes that field. Comments, key order, quoting and
1883
+ formatting survive, and so does every step not touched.
1884
+ - **An edited file is written with LF line endings.**
1885
+ - **Accelerator writes only against what it read.** If a file changed on disk while it was
1886
+ being edited, nothing is written, and the change is shown instead.
1887
+ - Code (`tests`, `before.script`) that Accelerator writes is a `|` block, even for one line.
1888
+
1889
+ When writing files yourself, especially from a program or an agent:
1890
+
1891
+ - Use two-space indentation and block style. Flow style (`[a, b]`, `{ secret: true }`)
1892
+ is fine for short values.
1893
+ - Quote any string that starts with `{`, `[`, `*`, `&`, `!`, `%`, `@` or `` ` ``, or
1894
+ that holds `: ` or ` #`.
1895
+ - Write JSON bodies and code as `|` blocks.
1896
+ - Keep `id` equal to the file name, and change both together.
1897
+ - Check the result against [Appendix A](#appendix-a-validation-rules).
1898
+
1899
+ ---
1900
+
1901
+ ## Appendix A. Validation rules
1902
+
1903
+ A file that breaks one of these rules is reported with a message naming the rule. A
1904
+ broken collection is shown as broken, and `ata` reports it as failed without running
1905
+ it. Rules checked at run time fail the step, or the run, before anything is sent.
1906
+
1907
+ **Every document**
1908
+
1909
+ - It must be YAML that parses, holding a mapping, in a file ending in `.yml`.
1910
+ - Unknown keys are errors in: a collection's top level, a step, `settings`, `before`,
1911
+ `body`, `project.yml`, `tls`, an environment file, an environment's `flags`, a param
1912
+ spec, `settings.yml` and `rules.yml`.
1913
+
1914
+ **Collections**
1915
+
1916
+ - `id` is present, equals the file name without `.yml`, and matches
1917
+ `^[A-Za-z0-9][A-Za-z0-9._-]*$`.
1918
+ - `id` is unique in its home (`collections/`, `requests/`, `bases/` or `endpoints/`),
1919
+ ignoring case.
1920
+ - There is no `name` key (use `id`).
1921
+ - A step has `tags` only if the collection has `stepTags: true`.
1922
+ - With `params`: no step is a use step or has `exclude`, and there is no `vars`, `setup`
1923
+ or `teardown`.
1924
+ - A `setup` or `teardown` step has no `tags` or `exclude`.
1925
+ - A data file, when present, reads (§2.8).
1926
+
1927
+ **Steps**
1928
+
1929
+ - A step has exactly one method key, out of `GET`, `POST`, `PUT`, `PATCH`, `DELETE`,
1930
+ `HEAD` and `OPTIONS`, in capitals, and its value is a string. A use step, a step
1931
+ reading a connection and a wait step have none.
1932
+ - A step has no key outside the table in §2.1.
1933
+ - A use step has no method key, `headers`, `body`, `settings` or `before`. `with` is used
1934
+ only with `use`.
1935
+ - `before` holds only `script`. `before.set` is not supported.
1936
+ - There is no `expect:` key; checks go in `tests`.
1937
+ - `base` is only ever `false`.
1938
+ - `forEach` is a string, only on a setup or teardown step, and a use step has none.
1939
+ - `useTests` is only ever `true`, only on a reusable requests file's step, and on one
1940
+ step at most.
1941
+ - A step reading a connection has no `headers`, `body`, `base` or `forEach`. A step
1942
+ opening one has no `forEach`, and a use step has no `connection` (§2.11).
1943
+ - A wait step's `wait` is a whole number from 0 to 600000, or a string holding a
1944
+ `{{variable}}`, and it holds nothing but `wait`, `name`, `docs`, `flags`, `tags` and
1945
+ `exclude`.
1946
+ At run time its `{{variables}}` must come to a whole number from 0 to 600000 (§2.1).
1947
+
1948
+ **Values**
1949
+
1950
+ - A variable value, in `vars`, `with` or a param, is a string, number, boolean or null.
1951
+ - A header is a string, `{ value, enabled?, description? }`, or a list of either.
1952
+ - A tag matches `^[A-Za-z0-9._:-]+$`.
1953
+ - A flag name matches `^[A-Za-z0-9_][A-Za-z0-9_.-]*$`, and a flag value is a string,
1954
+ number or boolean.
1955
+ - A connection name matches `^[A-Za-z0-9_][A-Za-z0-9_.-]*$`.
1956
+ - A `settings` value has its type in §2.3.
1957
+
1958
+ **Bodies**
1959
+
1960
+ - A body declares exactly one of `json`, `xml`, `text`, `form`, `multipart`, `graphql`
1961
+ and `file`.
1962
+ - `json`, `xml` and `text` are strings, and `form` is a map of strings.
1963
+ - A multipart field is a string, `{ value, contentType? }`,
1964
+ `{ file, contentType?, filename? }`, or a non-empty list of them.
1965
+ - `file` paths are relative.
1966
+
1967
+ **Library files**
1968
+
1969
+ - A reusable requests file (`requests/`) has `params`, uses no other, and has no
1970
+ `vars`.
1971
+ - A base collection (`bases/`) has no `steps`, `setup`, `teardown` or `params`, and no
1972
+ `extends`.
1973
+ - An endpoint (`endpoints/`) has a URL that is a path starting with `/`, and no use
1974
+ steps, connections or `exclude`. An endpoints file has no `setup` or `teardown`.
1975
+ - A check file's name is a JavaScript identifier; if it isn't, the file is not loaded.
1976
+ - `collections/`, `requests/`, `endpoints/` and `bases/` hold files at most one folder
1977
+ down.
1978
+
1979
+ **Portability (§1.2)**
1980
+
1981
+ - A name a file refers to is spelled as it is on disk, case included: a `use:` or
1982
+ `extends:` name, `uses`, a `tls.ca` file and a file a body sends.
1983
+ - The folders and files ata looks for in a project are lower case: `collections/`, not
1984
+ `Collections/`.
1985
+ - No two names in `collections/`, `requests/`, `endpoints/`, `bases/`, `environments/`
1986
+ or `checks/` differ only in case.
1987
+ - No file or folder name is one Windows refuses.
1988
+
1989
+ **Projects**
1990
+
1991
+ - `uses` and `tls.ca` entries are relative paths.
1992
+ - The folder `uses` names has a `project.yml`, and does not itself `uses` another.
1993
+ - Each `tls.ca` file exists and holds a PEM or DER certificate.
1994
+ - `settings.yml` exists for `ata`; a global project's is optional. The `environmentType`
1995
+ a run ends up with, if any, names an environment that exists.
1996
+ - In `rules.yml`, each rule has a value of the type §1.4 gives it, a pattern compiles,
1997
+ `tests.only` lists `ata`, and `guide` is a string. What breaks a rule is a finding of `ata lint`, never an
1998
+ error: it stops nothing (§1.4).
1999
+
2000
+ **At run time**
2001
+
2002
+ - Every `{{variable}}` resolves, with no loop and no chain deeper than 16.
2003
+ - Every secret has a value.
2004
+ - Every feature flag named is known to the run.
2005
+ - The flag `command`, if any, has every quote closed, starts, and exits `0` within 60
2006
+ seconds, printing a JSON object.
2007
+ - Every file a body names can be read from the project folder, or its global
2008
+ project's (§2.2).
2009
+ - Every `use:` and `extends:` names a usable file, and every `with:` suits its file.
2010
+ `ata get` checks these, and the body files named without `{{variables}}`, without
2011
+ running anything (§1.3).
2012
+ - A step's `forEach` resolves to a JSON array.
2013
+ - A step reading a connection finds it open: a step before it in the run opened it.
2014
+ - In `{{@name}}`, `name` holds text naming a variable that exists.
2015
+ - A `before.script` sets `req.body` to text, and only for a `json`, `xml`, `text` or
2016
+ `graphql` body; `req.headers` stays a map.
2017
+ - `ata.set`'s `scope`, when given, is `'run'`.
2018
+
2019
+ ---
2020
+
2021
+ ## Appendix B. A complete project
2022
+
2023
+ ```
2024
+ shop/
2025
+ ├── project.yml
2026
+ ├── settings.yml
2027
+ ├── collections/
2028
+ │ ├── health.yml
2029
+ │ └── users/
2030
+ │ ├── profile.yml
2031
+ │ ├── profile.csv
2032
+ │ └── avatar.yml
2033
+ ├── requests/
2034
+ │ └── login.yml
2035
+ ├── checks/
2036
+ │ └── common.js
2037
+ ├── files/
2038
+ │ └── avatar.png
2039
+ ├── environments/
2040
+ │ └── staging.yml
2041
+ └── .env apiKey=… (not committed)
2042
+ ```
2043
+
2044
+ ```yaml
2045
+ # project.yml
2046
+ name: Shop
2047
+ vars:
2048
+ apiVersion: '2'
2049
+ ```
2050
+
2051
+ ```yaml
2052
+ # settings.yml
2053
+ environmentType: staging
2054
+ limitConcurrency: 2
2055
+ generateJUnitResults: true
2056
+ ```
2057
+
2058
+ ```yaml
2059
+ # environments/staging.yml
2060
+ vars:
2061
+ baseUrl: https://staging.example.com
2062
+ adminUser: admin@example.com
2063
+ apiKey: { secret: true }
2064
+ flags:
2065
+ values: { avatars: true }
2066
+ ```
2067
+
2068
+ ```yaml
2069
+ # requests/login.yml
2070
+ id: login
2071
+ params:
2072
+ username: { required: true }
2073
+ steps:
2074
+ - name: log in
2075
+ POST: '{{baseUrl}}/v{{apiVersion}}/login'
2076
+ headers:
2077
+ X-Api-Key: '{{apiKey}}'
2078
+ body:
2079
+ json: '{ "user": "{{params.username}}" }'
2080
+ tests: |
2081
+ ata.expectResponseStatusCodeToBe(200)
2082
+ ata.expectResponseBodyToHaveProperty('token', 'token', 'setAsCollectionVariable')
2083
+ ```
2084
+
2085
+ ```js
2086
+ // checks/common.js
2087
+ export function expectJson() {
2088
+ ata.expectResponseToHaveHeader('Content-Type', /^application\/json/)
2089
+ }
2090
+ ```
2091
+
2092
+ ```yaml
2093
+ # collections/health.yml
2094
+ id: health
2095
+ tags: [smoke]
2096
+ steps:
2097
+ - name: service is up
2098
+ GET: '{{baseUrl}}/health'
2099
+ tests: |
2100
+ ata.expectResponseStatusCodeToBe(200)
2101
+ checks.common.expectJson()
2102
+ ```
2103
+
2104
+ ```yaml
2105
+ # collections/users/profile.yml
2106
+ id: profile
2107
+ steps:
2108
+ - use: login # sets token for the steps after it
2109
+ with: { username: '{{adminUser}}' }
2110
+
2111
+ - name: get user
2112
+ GET: '{{baseUrl}}/v{{apiVersion}}/users/{{userId}}'
2113
+ headers:
2114
+ Authorization: Bearer {{token}}
2115
+ tests: |
2116
+ ata.expectResponseStatusCodeToBe(ata.get('expectedStatus'))
2117
+ ```
2118
+
2119
+ ```csv
2120
+ userId,expectedStatus,iterationLabel
2121
+ 1001,200,Known user
2122
+ 9999,404,Unknown user
2123
+ ```
2124
+
2125
+ ```yaml
2126
+ # collections/users/avatar.yml
2127
+ id: avatar
2128
+ flags: { avatars: true } # runs only where the environment turns avatars on
2129
+ steps:
2130
+ - use: login
2131
+ with: { username: '{{adminUser}}' }
2132
+
2133
+ - name: upload avatar
2134
+ POST: '{{baseUrl}}/v{{apiVersion}}/users/1001/avatar'
2135
+ headers:
2136
+ Authorization: Bearer {{token}}
2137
+ body:
2138
+ multipart:
2139
+ caption: Profile photo
2140
+ image: { file: files/avatar.png }
2141
+ tests: |
2142
+ ata.expectResponseStatusCodeToBe(201)
2143
+ ```
2144
+
2145
+ `ata all` runs `health` and `avatar` once each, and `profile` twice: once for each row
2146
+ of `profile.csv`.