@schwabyio/ata 0.0.0-stage → 0.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +352 -2
- package/dist/FUNCTIONS.md +594 -0
- package/dist/SPEC.md +2146 -0
- package/dist/THIRD_PARTY_NOTICES.txt +97 -0
- package/dist/ata.js +1938 -0
- package/dist/chunks/chunk-4LUFMFJ5.js +234 -0
- package/dist/chunks/chunk-I3XKICH2.js +592 -0
- package/dist/chunks/chunk-IY7FUPUW.js +67973 -0
- package/dist/index.d.ts +3 -0
- package/dist/index.js +14 -0
- package/dist/playwright.d.ts +37 -0
- package/dist/playwright.js +183 -0
- package/dist/types.d.ts +255 -0
- package/dist/worker.js +14 -0
- package/package.json +72 -4
package/dist/SPEC.md
ADDED
|
@@ -0,0 +1,2146 @@
|
|
|
1
|
+
# The Accelerator file format
|
|
2
|
+
|
|
3
|
+
Version 0.20.0, the version of Accelerator and `ata` that reads it: the two are released
|
|
4
|
+
together.
|
|
5
|
+
|
|
6
|
+
This document specifies the YAML files that **Accelerator**, the desktop app, and **`ata`**,
|
|
7
|
+
the command-line runner, read and write. Together they make up Accelerator Test
|
|
8
|
+
Automation, a tool for testing HTTP APIs. A folder that follows this document works in
|
|
9
|
+
both of them.
|
|
10
|
+
|
|
11
|
+
It is written for people and for coding agents alike. Every key has a table saying its
|
|
12
|
+
type, whether it is required and its default. Every rule that makes a file invalid is
|
|
13
|
+
stated where it applies, and all of them are collected in
|
|
14
|
+
[Appendix A](#appendix-a-validation-rules). [Appendix B](#appendix-b-a-complete-project)
|
|
15
|
+
is a complete, valid project to start from.
|
|
16
|
+
|
|
17
|
+
**Conventions**
|
|
18
|
+
|
|
19
|
+
- **must**, **must not**, **should** and **may** are used as in RFC 2119.
|
|
20
|
+
- Every file is UTF-8 YAML 1.2, one document holding a mapping, with the extension
|
|
21
|
+
**`.yml`**. A `.yaml` file is not read.
|
|
22
|
+
- Paths written in files are relative, and use `/`.
|
|
23
|
+
- Section numbers are stable. Accelerator's error messages cite them, as in
|
|
24
|
+
"(SPEC.md §2.5)".
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## At a glance
|
|
29
|
+
|
|
30
|
+
| File | Where | What it is | § |
|
|
31
|
+
| ----------------------------- | --------------------------------------- | ------------------------------------------- | ---- |
|
|
32
|
+
| `collections/<id>.yml` | `collections/`, or one folder inside it | A collection: requests run in order | §2 |
|
|
33
|
+
| `collections/<id>.csv\|.json` | Beside its collection | A data file: the collection runs once a row | §2.8 |
|
|
34
|
+
| `environments/<name>.yml` | `environments/` | Variables, secrets and flags for one target | §6 |
|
|
35
|
+
| `project.yml` | The project folder | Name, global project, variables, trust | §1.1 |
|
|
36
|
+
| `settings.yml` | The project folder | How `ata` runs the project | §1.3 |
|
|
37
|
+
| `rules.yml` | The project folder | How the project's files are written | §1.4 |
|
|
38
|
+
| `requests/<id>.yml` | `requests/`, or one folder inside it | Reusable requests, run by `use:` | §2.5 |
|
|
39
|
+
| `endpoints/<id>.yml` | `endpoints/`, or one folder inside it | Defaults and checks per method and path | §2.6 |
|
|
40
|
+
| `bases/<id>.yml` | `bases/`, or one folder inside it | A base collection, for `extends:` | §2.7 |
|
|
41
|
+
| `checks/<name>.js` | `checks/` | Shared check functions | §5 |
|
|
42
|
+
| `.env` | The project folder, never committed | Values for secrets | §6 |
|
|
43
|
+
|
|
44
|
+
The smallest project `ata` runs is three files:
|
|
45
|
+
|
|
46
|
+
```yaml
|
|
47
|
+
# collections/health.yml
|
|
48
|
+
id: health
|
|
49
|
+
steps:
|
|
50
|
+
- name: service is up
|
|
51
|
+
GET: '{{baseUrl}}/health'
|
|
52
|
+
tests: |
|
|
53
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
```yaml
|
|
57
|
+
# environments/local.yml
|
|
58
|
+
vars:
|
|
59
|
+
baseUrl: http://localhost:8080
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
```yaml
|
|
63
|
+
# settings.yml: how ata runs the project (§1.3)
|
|
64
|
+
environmentType: local
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The desktop app needs only the first two, and the environment is picked in the app.
|
|
68
|
+
|
|
69
|
+
**The mistakes that come up most:**
|
|
70
|
+
|
|
71
|
+
1. **A value that starts with `{{` must be quoted.** YAML reads an unquoted `{` as the
|
|
72
|
+
start of a map, so `GET: {{baseUrl}}/x` is invalid. Write `GET: '{{baseUrl}}/x'`.
|
|
73
|
+
2. **A collection's `id` must equal its file name** without `.yml` (§2).
|
|
74
|
+
3. **Each step has exactly one method key**, in capitals, whose value is the URL: `GET:`,
|
|
75
|
+
not `get:` or `method: GET` (§2.1).
|
|
76
|
+
4. **A JSON body is a string**, not a YAML map: `json: |` followed by the JSON (§2.2).
|
|
77
|
+
5. **Code goes in `tests` and `before.script`** as a block (`|`). Checks are calls on `ata`
|
|
78
|
+
(§3, §5).
|
|
79
|
+
6. **`vars` hold plain values only**. Anything computed is set in `before.script` with
|
|
80
|
+
`ata.set` (§4).
|
|
81
|
+
7. **An unknown `{{variable}}` fails the step.** It is never sent as literal text (§4).
|
|
82
|
+
8. **Step tags need `stepTags: true`** on the collection (§2.4).
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## 1. Layout
|
|
87
|
+
|
|
88
|
+
A **project** is a folder holding `collections/` and, usually, `environments/`:
|
|
89
|
+
|
|
90
|
+
```
|
|
91
|
+
payments/ a project
|
|
92
|
+
├── project.yml optional (§1.1)
|
|
93
|
+
├── settings.yml how ata runs it (§1.3)
|
|
94
|
+
├── rules.yml how its files are written (§1.4)
|
|
95
|
+
├── collections/
|
|
96
|
+
│ ├── smoke.yml a collection
|
|
97
|
+
│ └── checkout/ a folder grouping collections: one level only
|
|
98
|
+
│ ├── sessions.yml
|
|
99
|
+
│ ├── sessions.csv its data file (§2.8)
|
|
100
|
+
│ └── refunds.yml
|
|
101
|
+
├── environments/
|
|
102
|
+
│ ├── local.yml
|
|
103
|
+
│ └── staging.yml
|
|
104
|
+
├── requests/ reusable requests (§2.5)
|
|
105
|
+
├── endpoints/ endpoint bases (§2.6)
|
|
106
|
+
├── bases/ base collections (§2.7)
|
|
107
|
+
├── checks/ check files (§5)
|
|
108
|
+
├── files/ anything a body uploads (§2.2); any name will do
|
|
109
|
+
└── .env secret values; not committed (§6)
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
**Every `.yml` file directly in `collections/`, or in a folder one level down, is a
|
|
113
|
+
collection.** Nothing else is. Discovery is exact: no other `.yml` in a repository is
|
|
114
|
+
mistaken for a collection. A file that does not parse is reported as a broken
|
|
115
|
+
collection, never skipped in silence.
|
|
116
|
+
|
|
117
|
+
- A folder inside a folder of `collections/` is reported as a problem and not
|
|
118
|
+
read. `requests/`, `endpoints/` and `bases/` are read to the same depth, and
|
|
119
|
+
`checks/` only at its top level.
|
|
120
|
+
- Names starting with `.` are ignored, as are the folders `node_modules`, `.git`,
|
|
121
|
+
`reports`, `test-results`, `out` and `dist`.
|
|
122
|
+
- Folders carry no configuration and need no file of their own. They group
|
|
123
|
+
collections for display and for running a group.
|
|
124
|
+
|
|
125
|
+
A project is any folder: the root of a repository, or one service of a monorepo. A
|
|
126
|
+
monorepo is several projects, one per service, and they can share a **global project**
|
|
127
|
+
(§1.1):
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
platform/ a repository, not itself a project
|
|
131
|
+
├── services/auth/ a project
|
|
132
|
+
│ ├── project.yml uses: ../../shared
|
|
133
|
+
│ ├── collections/login.yml
|
|
134
|
+
│ └── environments/local.yml
|
|
135
|
+
├── services/users/ a project
|
|
136
|
+
│ └── collections/users.yml
|
|
137
|
+
└── shared/ a global project
|
|
138
|
+
├── project.yml
|
|
139
|
+
└── environments/local.yml
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
A project reads nothing above its own folder except the global project it names.
|
|
143
|
+
|
|
144
|
+
### 1.1 `project.yml` and global projects
|
|
145
|
+
|
|
146
|
+
```yaml
|
|
147
|
+
name: Payments # shown instead of the folder name
|
|
148
|
+
uses: ../../shared # a global project, relative to this one
|
|
149
|
+
vars: # for every collection in the project
|
|
150
|
+
region: eu
|
|
151
|
+
tls:
|
|
152
|
+
ca: # certificate files to trust, besides the system's
|
|
153
|
+
- certs/company-root.pem
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
The file is optional, and so is every key in it. Any other key is an error.
|
|
157
|
+
|
|
158
|
+
| Key | Type | Default | Meaning |
|
|
159
|
+
| -------- | ---------------------- | ----------- | ----------------------------------------------------- |
|
|
160
|
+
| `name` | string | folder name | Display name. |
|
|
161
|
+
| `uses` | string (relative path) | none | A global project whose files this one shares (below). |
|
|
162
|
+
| `vars` | map of plain values | none | Variables for every collection in the project (§4). |
|
|
163
|
+
| `tls.ca` | list of relative paths | none | Certificate files that requests trust (below). |
|
|
164
|
+
|
|
165
|
+
**`uses`** names a **global project**: an ordinary project whose `project.yml`
|
|
166
|
+
variables, `environments/`, `requests/`, `endpoints/`, `bases/`, `checks/`,
|
|
167
|
+
`settings.yml` and `rules.yml` every project using it shares.
|
|
168
|
+
|
|
169
|
+
- It must be a relative path. An absolute path is refused, since it would only work on
|
|
170
|
+
one machine. Write it with `/`; `\` reads the same.
|
|
171
|
+
- The folder it names must hold a `project.yml`.
|
|
172
|
+
- A global project must not `uses` another: one level, no chains, no loops.
|
|
173
|
+
- An environment in the global project merges under the project's environment of the
|
|
174
|
+
same name, key by key, and the project's values win. An environment only the global
|
|
175
|
+
project has is available too.
|
|
176
|
+
- The global project's `settings.yml` lies under the project's the same way (§1.3), and
|
|
177
|
+
so does its `rules.yml` (§1.4).
|
|
178
|
+
- A global project's own `collections/` is **not** shared: a project using it never
|
|
179
|
+
sees or runs those collections. A global project needs none. It may have one for a
|
|
180
|
+
single purpose: **testing what it shares**. A collection there can `use:` its reusable
|
|
181
|
+
requests, `extends:` its bases and call its checks, against its own environments, so a
|
|
182
|
+
broken shared piece fails in one place, before every project relying on it does.
|
|
183
|
+
Those collections run only when the global project itself does, as a project in
|
|
184
|
+
Accelerator or with `ata` in its folder, such as its own CI job.
|
|
185
|
+
|
|
186
|
+
**`tls.ca`** lists certificate files that requests trust, for a server whose certificate
|
|
187
|
+
a company or local CA signed, or a server's own self-signed certificate. A request
|
|
188
|
+
always trusts:
|
|
189
|
+
|
|
190
|
+
1. Node's bundled Mozilla roots, and any in `NODE_EXTRA_CA_CERTS`.
|
|
191
|
+
2. The operating system's trust store: the macOS Keychain, the Windows certificate
|
|
192
|
+
store, or the Linux CA bundle. A CA that IT installed, or that `mkcert -install`
|
|
193
|
+
added, is trusted with nothing written here.
|
|
194
|
+
3. `tls.ca`: the project's own files, then its global project's.
|
|
195
|
+
|
|
196
|
+
- Each entry is a relative path from the `project.yml` that lists it. An absolute path
|
|
197
|
+
is refused.
|
|
198
|
+
- A file is PEM (one certificate or a bundle) or a single DER certificate, such as a
|
|
199
|
+
`.cer` exported on Windows. A CA's certificate is public and safe to commit. A private
|
|
200
|
+
key never belongs here.
|
|
201
|
+
- A file that is missing or holds no certificate is a problem on the project. Accelerator
|
|
202
|
+
sends nothing from the project until it is fixed, and `ata` will not start.
|
|
203
|
+
- `tls.ca` only adds trust. Host names and expiry are still checked, and verification
|
|
204
|
+
is never turned off.
|
|
205
|
+
|
|
206
|
+
### 1.2 Portability
|
|
207
|
+
|
|
208
|
+
Projects are shared between macOS, Windows and Linux, and read the same on all three:
|
|
209
|
+
|
|
210
|
+
- Every path written in a file uses `/`. A `\` reads the same, but is never written.
|
|
211
|
+
- **Names match exactly, case included.** macOS and Windows find `requests/Auth/login.yml`
|
|
212
|
+
when the file is `requests/auth/login.yml`; Linux does not, so a project that works on a
|
|
213
|
+
laptop would fail in CI. Every name a file refers to must be spelled as it is on disk: a
|
|
214
|
+
`use:` or `extends:` name, `uses`, a `tls.ca` file, a file a body sends, and an
|
|
215
|
+
environment's file name. One that differs only in case is an error on every platform,
|
|
216
|
+
naming the spelling on disk. The folders and files of §1 are lower case.
|
|
217
|
+
- **No two names in one folder may differ only in case**, such as `Checkout/` and
|
|
218
|
+
`checkout/`: Linux can hold both, but a macOS or Windows checkout only one. This
|
|
219
|
+
applies in `collections/`, `requests/`, `endpoints/`, `bases/`, `environments/` and
|
|
220
|
+
`checks/`. Collection ids are unique ignoring case too (§2).
|
|
221
|
+
- File, folder and environment names must avoid what Windows refuses: the characters
|
|
222
|
+
`< > : " / \ | ? *`, control characters, a trailing dot or space, and the names `CON`,
|
|
223
|
+
`PRN`, `AUX`, `NUL`, `CONIN$`, `CONOUT$`, `COM1`–`COM9` and `LPT1`–`LPT9`, with or
|
|
224
|
+
without an extension. A file with such a name, made on macOS or Linux, is reported.
|
|
225
|
+
- Files should use LF line endings. Accelerator writes LF, and can add a `.gitattributes`
|
|
226
|
+
scoped to the project's own files so that git keeps them LF on every platform. The same
|
|
227
|
+
block keeps `files/` exactly as committed, so a body sends the same bytes everywhere. A
|
|
228
|
+
global project is a project too: its own block covers its `files/`. For files a body
|
|
229
|
+
sends from another folder, add a `-text` line of your own.
|
|
230
|
+
- The process environment is read by exact name on every platform, although Windows
|
|
231
|
+
itself ignores case (§4, §6).
|
|
232
|
+
|
|
233
|
+
### 1.3 `settings.yml`
|
|
234
|
+
|
|
235
|
+
How `ata` runs the project. It sits beside `collections/`, is committed, and `ata`
|
|
236
|
+
refuses to run without it. The desktop app does not read it.
|
|
237
|
+
|
|
238
|
+
```yaml
|
|
239
|
+
environmentType: staging # environments/<name>.yml
|
|
240
|
+
limitConcurrency: 4
|
|
241
|
+
timeoutCollection: 3600000
|
|
242
|
+
bail: false
|
|
243
|
+
tags: [smoke]
|
|
244
|
+
notTags: [slow]
|
|
245
|
+
generateJUnitResults: true
|
|
246
|
+
generateJsonResults: true
|
|
247
|
+
generateHtmlResults: true
|
|
248
|
+
autoOpenTestResultHtml: false
|
|
249
|
+
testResultsBasePath: test-results
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
Every key is optional. **A key not listed here is an error**, so a typo cannot quietly
|
|
253
|
+
run with a default.
|
|
254
|
+
|
|
255
|
+
| Key | Type | Default | Meaning |
|
|
256
|
+
| ------------------------ | ------------ | -------------- | --------------------------------------------------------------------------- |
|
|
257
|
+
| `environmentType` | string | none | The environment to run against, by name. It must exist (§6). |
|
|
258
|
+
| `limitConcurrency` | integer ≥ 1 | `1` | Collections run at once. Steps within a collection always run in order. |
|
|
259
|
+
| `timeoutCollection` | integer ≥ 1 | `3600000` | Stop a collection after this many ms and fail it; its teardown still runs. |
|
|
260
|
+
| `bail` | boolean | `false` | Stop a collection at its first failing step; the rest are reported skipped. |
|
|
261
|
+
| `tags` | list of tags | `[]` | Run only what these select (§2.4). Empty runs everything. |
|
|
262
|
+
| `notTags` | list of tags | `[]` | Leave out what these name (§2.4). |
|
|
263
|
+
| `generateJUnitResults` | boolean | `false` | Write `<testResultsBasePath>/junit/junit.xml`. |
|
|
264
|
+
| `generateJsonResults` | boolean | `false` | Write `<testResultsBasePath>/json/results.json`. |
|
|
265
|
+
| `generateHtmlResults` | boolean | `false` | Write `<testResultsBasePath>/html/summary.html` and a page per collection. |
|
|
266
|
+
| `autoOpenTestResultHtml` | boolean | `false` | Write the HTML report and open it when the run ends. |
|
|
267
|
+
| `testResultsBasePath` | string | `test-results` | Where reports go: relative to the project folder, or absolute. |
|
|
268
|
+
|
|
269
|
+
The results folder is emptied before every run. `ata` refuses to empty one that holds
|
|
270
|
+
files it did not write, or one that holds the project itself. Every report replaces
|
|
271
|
+
each secret's value with `[secret: NAME]` (§6).
|
|
272
|
+
|
|
273
|
+
**From the global project.** A global project (§1.1) can have a `settings.yml` too.
|
|
274
|
+
Every project that `uses` it runs with those settings under its own: a key the
|
|
275
|
+
project's file sets wins, and a key it leaves out comes from the global project's file.
|
|
276
|
+
|
|
277
|
+
- There is no switch to turn this off. A project undoes a shared setting by setting it
|
|
278
|
+
in its own file, back to the default if need be: `environmentType: null`, `tags: []`.
|
|
279
|
+
- A project still needs a `settings.yml` of its own, even an empty one. A global project
|
|
280
|
+
need not have one.
|
|
281
|
+
- A relative `testResultsBasePath` is relative to the project being run, whichever file
|
|
282
|
+
set it.
|
|
283
|
+
|
|
284
|
+
**Overrides.** Each setting can be overridden by an environment variable, then by a
|
|
285
|
+
command-line flag: `ATA_LIMIT_CONCURRENCY=8`, then `--limitConcurrency 8`. A list is
|
|
286
|
+
comma-separated: `--tags smoke,api`. In all, from lowest to highest: the default, the
|
|
287
|
+
global project's `settings.yml`, the project's, the environment variable, the flag.
|
|
288
|
+
`ata get` lists each setting that is not a default, and where it came from.
|
|
289
|
+
|
|
290
|
+
**Running.** `ata` runs from the project folder:
|
|
291
|
+
|
|
292
|
+
| Command | Runs |
|
|
293
|
+
| --------------------- | -------------------------------------------------------------------- |
|
|
294
|
+
| `ata get` | Nothing. It lists what `ata all` would run. |
|
|
295
|
+
| `ata all` | Every collection and step, except those with `exclude: true` (§2.4). |
|
|
296
|
+
| `ata smoke,checkout/` | The collections and folders named, in that order. |
|
|
297
|
+
|
|
298
|
+
`ata get` also reports each `use:` and `extends:` that would stop a run, and each file a
|
|
299
|
+
body names that is in neither the project nor its global project (§2.2), in every
|
|
300
|
+
collection, including those `ata all` leaves out (Appendix A). A file path with
|
|
301
|
+
`{{variables}}` is left to the run. It exits `1` when it finds one, or a broken
|
|
302
|
+
collection, and `0` otherwise.
|
|
303
|
+
|
|
304
|
+
A collection is named by its `id`, or by its place (`checkout/sessions`). A folder is
|
|
305
|
+
named by its name, and `checkout/` names only the folder. `--flag name=value` sets a
|
|
306
|
+
feature flag (§2.9), and `--json` prints the results as JSON. The exit code is `0` when
|
|
307
|
+
everything passed, `1` when something failed, and `2` when `ata` could not run.
|
|
308
|
+
|
|
309
|
+
### 1.4 `rules.yml`
|
|
310
|
+
|
|
311
|
+
How a project's files are named, laid out and written, so that everyone working on it,
|
|
312
|
+
people and coding agents alike, keeps it consistent. It sits beside `project.yml`, is
|
|
313
|
+
committed, and is optional. `ata lint` checks the project against it. Accelerator, the
|
|
314
|
+
desktop app, marks each file, folder and step that breaks a rule, and will not create or
|
|
315
|
+
rename a file or folder that would.
|
|
316
|
+
|
|
317
|
+
```yaml
|
|
318
|
+
ids:
|
|
319
|
+
collections: kebab-case # or a pattern, such as '{folder}-[a-z0-9-]+'
|
|
320
|
+
requests: camelCase
|
|
321
|
+
layout:
|
|
322
|
+
folders: required # every collection in a folder of collections/
|
|
323
|
+
folderNames: [accounts, payments, smoke, payments-errors]
|
|
324
|
+
kinds:
|
|
325
|
+
negative: # tagged negative: in a folder ending in -errors, which hold no other
|
|
326
|
+
tags: [negative]
|
|
327
|
+
folders: .*-errors
|
|
328
|
+
maxSteps: 30
|
|
329
|
+
steps:
|
|
330
|
+
names: required
|
|
331
|
+
url: ^\{\{baseUrl\}\} # no host written out
|
|
332
|
+
docs:
|
|
333
|
+
collections: required
|
|
334
|
+
tags:
|
|
335
|
+
allowed: [smoke, regression, slow]
|
|
336
|
+
tests:
|
|
337
|
+
only: [ata] # tests call the ata.* functions, and nothing else
|
|
338
|
+
statusCode: required
|
|
339
|
+
guide: |
|
|
340
|
+
Name each step for what it proves. Logins go through `requests/login`.
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
**Rules never change a run.** A file that breaks one opens, sends and runs as before.
|
|
344
|
+
`ata lint` reports it, and a CI job running `ata lint` fails on it. Accelerator marks it, and
|
|
345
|
+
flags what `tests.only` does not allow as a script is typed.
|
|
346
|
+
|
|
347
|
+
| Rule | Value | What it checks |
|
|
348
|
+
| -------------------- | ---------------------- | ------------------------------------------------------------------------------- |
|
|
349
|
+
| `ids.collections` | style or pattern | Each collection's id, its file name (§2). |
|
|
350
|
+
| `ids.requests` | style or pattern | Each reusable requests file's id (§2.5). |
|
|
351
|
+
| `ids.bases` | style or pattern | Each base collection's id (§2.7). |
|
|
352
|
+
| `ids.endpoints` | style or pattern | Each endpoints file's id (§2.6). |
|
|
353
|
+
| `layout.folders` | `required` | Every collection sits in a folder of `collections/`, none at its top. |
|
|
354
|
+
| `layout.folderNames` | list, style or pattern | Each folder of `collections/`: one of the list, or following the format. |
|
|
355
|
+
| `layout.kinds` | map of kinds (below) | Each kind of collection sits in its own folders, and they hold no other. |
|
|
356
|
+
| `layout.maxSteps` | integer ≥ 1 | The most steps a collection has in `steps`; `setup` and `teardown` aside. |
|
|
357
|
+
| `layout.bases` | `none` | No base collections: no file in `bases/`, and no `extends:` (below). |
|
|
358
|
+
| `layout.checks` | `none` | No check files: no `.js` file in `checks/`, and no use of `checks` (below). |
|
|
359
|
+
| `steps.names` | `required` | Every step has a `name`, and no other step of its file has the same one. |
|
|
360
|
+
| `steps.url` | pattern | Every request step's URL, as written, matches it (below). |
|
|
361
|
+
| `docs.collections` | `required` | Every collection has `docs`. |
|
|
362
|
+
| `docs.requests` | `required` | Every reusable requests file has `docs`. |
|
|
363
|
+
| `docs.steps` | `required` | Every step of a collection or reusable requests file has `docs`, in every list. |
|
|
364
|
+
| `tags.allowed` | list of tags | Every tag on a collection or a step is one of these. |
|
|
365
|
+
| `tags.collections` | `required` | Every collection has `tags` of its own. |
|
|
366
|
+
| `tests.only` | list (below) | What a `tests` script may call. |
|
|
367
|
+
| `tests.everyStep` | `required` | Every step that sends a request or reads a connection is checked (below). |
|
|
368
|
+
| `tests.statusCode` | `required` | Every request step's checks include its status code (below). |
|
|
369
|
+
|
|
370
|
+
- Any other group or rule is an error, as is a value of the wrong type.
|
|
371
|
+
- A rule that takes `required` also takes `optional`, which turns it off, and one that
|
|
372
|
+
takes `none` takes `allowed`.
|
|
373
|
+
- A **style** is one of `kebab-case` (`create-user`), `snake_case` (`create_user`),
|
|
374
|
+
`camelCase` (`createUser`) and `PascalCase` (`CreateUser`). Lower case, in the first two,
|
|
375
|
+
includes digits: `404-handling` is kebab-case.
|
|
376
|
+
- A **pattern** is a JavaScript regular expression that the whole name must match, as if
|
|
377
|
+
it began with `^` and ended with `$`. `{folder}` in it stands for the folder the file is
|
|
378
|
+
in, so `{folder}-[a-z0-9-]+` asks for `collections/payments/payments-refunds.yml`. A
|
|
379
|
+
value that is not a style and has none of a pattern's characters (`^`, `$`, `.`, `*`,
|
|
380
|
+
`+`, `?`, `(`, `)`, `[`, `]`, `{`, `}`, `|` or a backslash) is an error, so a
|
|
381
|
+
misspelled style is not taken for a pattern.
|
|
382
|
+
- Rules check the project's own files, in `collections/`, `requests/`, `bases/` and
|
|
383
|
+
`endpoints/`, and `layout.checks` its `checks/`. A global project's files follow the
|
|
384
|
+
global project's `rules.yml`, checked when `ata lint` runs in its folder.
|
|
385
|
+
- `steps.*`, `docs.steps`, `tests.everyStep` and `tests.statusCode` are about the steps of
|
|
386
|
+
collections and reusable requests files. An endpoint is a method and a path pattern,
|
|
387
|
+
not a step that runs, so they leave endpoints files alone.
|
|
388
|
+
|
|
389
|
+
**`steps.url`** is a JavaScript regular expression that each request step's URL, as
|
|
390
|
+
written with its `{{variables}}`, must match. Unlike an id pattern it is not anchored:
|
|
391
|
+
`^\{\{baseUrl\}\}` asks that every URL start with `{{baseUrl}}`, so no host is written
|
|
392
|
+
out in a step. A use step and a step reading a connection have no URL of their own.
|
|
393
|
+
|
|
394
|
+
**`tests.everyStep` and `tests.statusCode`** count every script that runs after a step's
|
|
395
|
+
response, wherever it is written: the step's own `tests`, its file's, its base
|
|
396
|
+
collection's (§2.7) and its endpoint's (§2.6), unless the step has `base: false`. With
|
|
397
|
+
`tests.statusCode`, one of them calls `ata.expectResponseStatusCodeToBe`, in any branch,
|
|
398
|
+
or calls a check function (§5) whose own code does. A use step is left to its reusable
|
|
399
|
+
requests file, whose steps are checked there. A step reading a connection needs tests
|
|
400
|
+
under `tests.everyStep`, but has no status code of its own to check.
|
|
401
|
+
|
|
402
|
+
**`tests.only`** lists what a `tests` script may call. It must list `ata`:
|
|
403
|
+
|
|
404
|
+
| Entry | Allows |
|
|
405
|
+
| ---------- | --------------------------------------------------------- |
|
|
406
|
+
| `ata` | The `ata.*` functions (§5), except `ata.test`. |
|
|
407
|
+
| `ata.test` | Named checks of your own, with any code inside them (§5). |
|
|
408
|
+
| `checks` | The project's check files, `checks.<file>.<function>()`. |
|
|
409
|
+
| `console` | `console.log()` and the rest. |
|
|
410
|
+
|
|
411
|
+
With `only: [ata]`, a `tests` script holds calls to `ata.*` functions and nothing else:
|
|
412
|
+
|
|
413
|
+
- **Every argument is a value**: a string, number, `true`, `false`, `null` or regular
|
|
414
|
+
expression, a template string, a list or object of values, or `new RegExp(…)` of
|
|
415
|
+
values. It may read `res`, `req`, `params`, `endpoint` and `item` (§5), call another
|
|
416
|
+
`ata.*` function such as `ata.get('id')`, or `JSON.parse` a value, since `ata.set` keeps
|
|
417
|
+
a list or object as JSON text.
|
|
418
|
+
- **An `if` may choose by feature flag** (§2.9): its condition is `ata.flag()` calls
|
|
419
|
+
compared with values, joined by `!`, `&&` and `||`, and what it holds follows the same
|
|
420
|
+
rules.
|
|
421
|
+
- **Nothing else**: no variables, loops, functions of your own, `assert`, `checks.*` or
|
|
422
|
+
`ata.test`. This is checked on the script's syntax, so a call cannot be hidden by
|
|
423
|
+
renaming it (`const c = checks`).
|
|
424
|
+
|
|
425
|
+
```yaml
|
|
426
|
+
tests: |
|
|
427
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
428
|
+
ata.expectResponseBodyToHaveProperty('owner', ata.get('userId'))
|
|
429
|
+
ata.set('nextPage', res.body.links.next)
|
|
430
|
+
if (ata.flag('newCheckout') === true) {
|
|
431
|
+
ata.expectResponseBodyToHaveProperty('version', 2)
|
|
432
|
+
}
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
`tests.only` checks every `tests` script: a collection's own and each of its steps', and
|
|
436
|
+
those of reusable requests files, base collections and endpoints files.
|
|
437
|
+
`before.script` is not checked.
|
|
438
|
+
|
|
439
|
+
**`layout.kinds`** keeps each kind of collection in its own folders: negative tests in
|
|
440
|
+
the `-errors` folders, say, and nothing else there.
|
|
441
|
+
|
|
442
|
+
```yaml
|
|
443
|
+
layout:
|
|
444
|
+
kinds:
|
|
445
|
+
negative:
|
|
446
|
+
tags: [negative] # a collection tagged negative is one
|
|
447
|
+
id: .*_N # and so is one whose id ends in _N
|
|
448
|
+
folders: .*-errors # it goes in a folder ending in -errors, and those hold no other
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
- Each kind has a name of your choosing, used in its findings, and three keys:
|
|
452
|
+
- **`tags`**, a list: a collection with any of these tags of its own is one. A step's
|
|
453
|
+
tags do not count.
|
|
454
|
+
- **`id`**, a style or pattern: so is a collection whose id follows it. Like an id
|
|
455
|
+
pattern, it matches the whole id, so `.*_N` is an id ending in `_N`.
|
|
456
|
+
- **`folders`**, required: a list of folder names, a style or a pattern, as
|
|
457
|
+
`layout.folderNames` takes. A pattern matches the whole name, so `.*-errors` is a
|
|
458
|
+
folder ending in `-errors`, while `-errors$` matches only a folder called `-errors`.
|
|
459
|
+
- A kind needs `tags`, `id` or both. With both, either makes a collection one.
|
|
460
|
+
- A kind's `id` does not loosen `ids.collections`. An id ending in `_N` is neither
|
|
461
|
+
kebab-case nor snake_case, so a project with both sets a pattern that allows it, such
|
|
462
|
+
as `ids: { collections: '[a-z0-9_]+(_N)?' }`.
|
|
463
|
+
- **One of a kind outside its folders** is a finding, at the tag that makes it one, or
|
|
464
|
+
at its `id` when the id does. A collection at the top of `collections/` is in no folder.
|
|
465
|
+
- **One not of the kind inside them** is a finding too. So collections without the tag
|
|
466
|
+
or the id, the positive tests here, stay out of the `-errors` folders.
|
|
467
|
+
- Each kind is checked on its own, so a project can have several, such as `slow` in
|
|
468
|
+
`slow/` beside `negative` in `.*-errors`.
|
|
469
|
+
- Accelerator will not rename, move or copy a collection, or rename a folder, where it
|
|
470
|
+
would break this. A new collection may be made in a kind's folders when the kind has
|
|
471
|
+
`tags`, since it can only be tagged once it exists. Until it is, it is a finding.
|
|
472
|
+
- A project's `layout.kinds` replaces its global project's whole, as every rule's value
|
|
473
|
+
does. `null` turns it off.
|
|
474
|
+
|
|
475
|
+
**`layout.bases` and `layout.checks`** keep a project to what its own files say, with no
|
|
476
|
+
base collection (§2.7) or check file (§5) adding to them:
|
|
477
|
+
|
|
478
|
+
```yaml
|
|
479
|
+
layout:
|
|
480
|
+
bases: none
|
|
481
|
+
checks: none
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
- **`layout.bases: none`**: each file in `bases/` is a finding, and so is every
|
|
485
|
+
`extends:`, whether it names one of the project's bases or one of its global project's
|
|
486
|
+
(`global:name`). Accelerator offers no new base collection, and no base to choose for a
|
|
487
|
+
collection that has none.
|
|
488
|
+
- **`layout.checks: none`**: each `.js` file in `checks/` is a finding, and so is every
|
|
489
|
+
use of `checks` in a script: `before.script` and `tests`, the file's own and each
|
|
490
|
+
step's, in every file the rules check. A use is any reference to `checks`, so
|
|
491
|
+
`const c = checks` is one too. It is a finding whichever project the check file is in.
|
|
492
|
+
- With `tests.only` too, a `checks.*` call in `tests` breaks both rules, but it is
|
|
493
|
+
reported once, under `layout.checks`.
|
|
494
|
+
- To keep every project that shares a global project from them, set them in the global
|
|
495
|
+
project's `rules.yml`. A project can still turn either off in its own, with `allowed`
|
|
496
|
+
or `null`, and that shows in its diff and in `ata rules`.
|
|
497
|
+
|
|
498
|
+
**`guide`** is Markdown, for what no rule can check: how steps are named, which request
|
|
499
|
+
set a login goes through, what a collection is for. `ata rules` prints it after the
|
|
500
|
+
rules, and Accelerator shows it in Project settings. Each file's guide is read, the global
|
|
501
|
+
project's first, so a project adds to a shared guide rather than replacing it.
|
|
502
|
+
|
|
503
|
+
**From the global project.** A global project (§1.1) can have a `rules.yml` too, and it
|
|
504
|
+
lies under the project's, rule by rule. There is no switch to turn this off: a project
|
|
505
|
+
changes a shared rule by setting it in its own file, and turns it off with `null`, or
|
|
506
|
+
`optional` for a `required` rule. A group set to `null`, such as `tags: null`, turns off
|
|
507
|
+
every rule in it. Neither file is required.
|
|
508
|
+
|
|
509
|
+
**Checking.** From a project folder, or a global project's:
|
|
510
|
+
|
|
511
|
+
| Command | Does |
|
|
512
|
+
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
513
|
+
| `ata lint` | Checks every file against the rules, and prints each finding at its file and line, with its rule and the file the rule came from. Exits `1` on a finding. |
|
|
514
|
+
| `ata rules` | Lists each rule set, its value, the file it came from and what it means, rules turned off included, then the guide. |
|
|
515
|
+
|
|
516
|
+
- Neither needs `settings.yml` or runs anything, and both take `--json`.
|
|
517
|
+
- A file that will not parse is a finding, since no rule could be checked in it.
|
|
518
|
+
- Both exit `2` when a rule is not valid, or `project.yml` names a global project that
|
|
519
|
+
cannot be used: its rules would quietly go unchecked.
|
|
520
|
+
- A coding agent can run `ata rules` to learn a project's conventions before writing, and
|
|
521
|
+
`ata lint --json` after, to fix what it finds. Accelerator offers to add a section saying so
|
|
522
|
+
to the project's `AGENTS.md`, which coding agents read; nothing is written until asked.
|
|
523
|
+
|
|
524
|
+
---
|
|
525
|
+
|
|
526
|
+
## 2. Collection file
|
|
527
|
+
|
|
528
|
+
A **collection** is one file holding an ordered list of requests, called **steps**. Its
|
|
529
|
+
steps run in list order and share one variable scope, so what one step captures, the
|
|
530
|
+
next can use.
|
|
531
|
+
|
|
532
|
+
```yaml
|
|
533
|
+
# collections/checkout.yml
|
|
534
|
+
id: checkout
|
|
535
|
+
docs: |
|
|
536
|
+
Create a session, capture it, then read it back.
|
|
537
|
+
tags: [smoke]
|
|
538
|
+
headers: # sent with every step
|
|
539
|
+
Accept: application/json
|
|
540
|
+
settings:
|
|
541
|
+
timeout: 10000
|
|
542
|
+
vars:
|
|
543
|
+
apiVersion: '2'
|
|
544
|
+
|
|
545
|
+
steps:
|
|
546
|
+
- name: create session
|
|
547
|
+
POST: '{{baseUrl}}/v{{apiVersion}}/sessions'
|
|
548
|
+
body:
|
|
549
|
+
json: |
|
|
550
|
+
{ "amount": 1200 }
|
|
551
|
+
tests: |
|
|
552
|
+
ata.expectResponseStatusCodeToBe(201)
|
|
553
|
+
ata.expectResponseBodyToHaveProperty('id', 'sessionId', 'setAsCollectionVariable')
|
|
554
|
+
|
|
555
|
+
- name: read it back
|
|
556
|
+
GET: '{{baseUrl}}/v{{apiVersion}}/sessions/{{sessionId}}'
|
|
557
|
+
tests: |
|
|
558
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
559
|
+
ata.expectResponseBodyToHaveProperty('amount', 1200)
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
### Collection keys
|
|
563
|
+
|
|
564
|
+
Any key not listed here is an error.
|
|
565
|
+
|
|
566
|
+
| Key | Type | Required | Default | Meaning |
|
|
567
|
+
| ---------- | ------------------- | -------- | ------- | ----------------------------------------------------------------------------------------- |
|
|
568
|
+
| `id` | string | **yes** | | The file name without `.yml` (below). |
|
|
569
|
+
| `steps` | list of steps | no | `[]` | The requests, in run order (§2.1). |
|
|
570
|
+
| `setup` | list of steps | no | | Run once before `steps`; what it sets lasts the run (§2.10). |
|
|
571
|
+
| `teardown` | list of steps | no | | Run once after the rest, even when a step failed (§2.10). |
|
|
572
|
+
| `docs` | string | no | | Markdown (below). |
|
|
573
|
+
| `tags` | list of tags | no | | Tags that select the whole collection (§2.4). |
|
|
574
|
+
| `stepTags` | boolean | no | `false` | `true` lets steps carry their own tags (§2.4). |
|
|
575
|
+
| `exclude` | boolean | no | `false` | `true` leaves it out of group runs (§2.4). |
|
|
576
|
+
| `flags` | map | no | | Feature flags the whole collection needs (§2.9). |
|
|
577
|
+
| `headers` | map | no | | Sent with every step; a step's own header of the same name wins (§2.3). |
|
|
578
|
+
| `settings` | map | no | | Defaults for every step (§2.3). |
|
|
579
|
+
| `vars` | map of plain values | no | | Collection variables (§4). |
|
|
580
|
+
| `before` | map | no | | `script:` run before every step (§5). |
|
|
581
|
+
| `tests` | string | no | | JavaScript run after every step, before the step's own (§5). |
|
|
582
|
+
| `extends` | string | no | | A base collection to build on (§2.7). |
|
|
583
|
+
| `params` | map | no | | The inputs it takes as a reusable requests file (§2.5). Only in `requests/`, in practice. |
|
|
584
|
+
|
|
585
|
+
A collection has no `name` key. Its `id` is its name, and a file with `name:` is
|
|
586
|
+
rejected with a message saying so.
|
|
587
|
+
|
|
588
|
+
### `id`
|
|
589
|
+
|
|
590
|
+
Every collection file must say its **id**, which is its file name without `.yml`,
|
|
591
|
+
exactly. `collections/checkout/sessions.yml` starts `id: sessions`.
|
|
592
|
+
|
|
593
|
+
- **It must match the file name.** A file whose `id` is missing or different is a
|
|
594
|
+
broken collection. Renaming a file means changing its `id` too.
|
|
595
|
+
- **It must be unique in its home, ignoring case.** No two files in a project's
|
|
596
|
+
`collections/` may share an id, including files in different folders of it; the
|
|
597
|
+
same holds for `requests/`, `bases/` and `endpoints/`. Case is ignored because macOS
|
|
598
|
+
and Windows file systems ignore it. Every file sharing an id is broken.
|
|
599
|
+
- **It is letters, digits and `- _ .`, starting with a letter or digit**:
|
|
600
|
+
`^[A-Za-z0-9][A-Za-z0-9._-]*$`. It is also typed on a command line.
|
|
601
|
+
|
|
602
|
+
Because an id is unique, it names the collection everywhere on its own: in the app, on
|
|
603
|
+
`ata`'s command line and in reports.
|
|
604
|
+
|
|
605
|
+
### `docs`
|
|
606
|
+
|
|
607
|
+
A collection's or a step's `docs` is Markdown, shown in the desktop app and in `ata`'s
|
|
608
|
+
HTML report: headings, paragraphs, lists, tables, fenced code, block quotes, rules, and
|
|
609
|
+
inline code, emphasis and links (http and https only). HTML is shown as the text it is,
|
|
610
|
+
never rendered. Anything else stays the text it was written as.
|
|
611
|
+
|
|
612
|
+
GitHub's callouts are a block quote whose first line is the callout's name, and are shown
|
|
613
|
+
with its colour, icon and title:
|
|
614
|
+
|
|
615
|
+
```markdown
|
|
616
|
+
> [!WARNING]
|
|
617
|
+
> The cancel step needs the `orders:write` scope.
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
The names are `NOTE`, `TIP`, `IMPORTANT`, `WARNING` and `CAUTION`, in any case.
|
|
621
|
+
|
|
622
|
+
Some text is coloured:
|
|
623
|
+
|
|
624
|
+
- `{{variables}}` are marked, in code or not, as the app's URL bar marks them.
|
|
625
|
+
- Inline code that starts with a method, such as `` `GET /orders` ``, shows the method in
|
|
626
|
+
its colour. Inline code that is a status code, such as `` `201` `` or
|
|
627
|
+
`` `404 Not Found` `` or `` `5xx` ``, is shown in its class's colour.
|
|
628
|
+
- Fenced code is coloured when its fence names `json` or `xml`. The app also colours
|
|
629
|
+
`js` and `ts`.
|
|
630
|
+
|
|
631
|
+
### 2.1 Steps
|
|
632
|
+
|
|
633
|
+
**List order is run order.** A step is a request: it carries **exactly one HTTP method
|
|
634
|
+
key**, in capitals, whose value is the URL as a string.
|
|
635
|
+
|
|
636
|
+
```yaml
|
|
637
|
+
- name: create session
|
|
638
|
+
POST: '{{baseUrl}}/sessions?source=api'
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
Methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`.
|
|
642
|
+
|
|
643
|
+
| Key | Type | Required | Meaning |
|
|
644
|
+
| ------------ | ------------ | -------- | ------------------------------------------------------------------------------- |
|
|
645
|
+
| `<METHOD>` | string | **yes** | The URL, query string included. |
|
|
646
|
+
| `name` | string | no | Display name. Defaults to the method and URL. |
|
|
647
|
+
| `headers` | map | no | Over the collection's headers (§2.3). |
|
|
648
|
+
| `body` | map | no | Exactly one kind of body (§2.2). |
|
|
649
|
+
| `settings` | map | no | Over the collection's settings (§2.3). |
|
|
650
|
+
| `before` | map | no | `script:` run before the request (§5). |
|
|
651
|
+
| `tests` | string | no | The checks, as JavaScript: calls on `ata` (§3) and any other code (§5). |
|
|
652
|
+
| `tags` | list of tags | no | The step's own tags. Only with `stepTags: true` on the collection (§2.4). |
|
|
653
|
+
| `exclude` | boolean | no | `true` leaves it out of group runs. Only on a step of `steps` (§2.4). |
|
|
654
|
+
| `flags` | map | no | Feature flags the step needs (§2.9). |
|
|
655
|
+
| `forEach` | string | no | In setup or teardown: send the request once for each item of a list (below). |
|
|
656
|
+
| `useTests` | `true` | no | In a reusable requests file: the use step's `tests` check this response (§2.5). |
|
|
657
|
+
| `connection` | string | no | Keep the event stream this request opens as a connection (§2.11). |
|
|
658
|
+
| `base` | `false` | no | `false` leaves the step's endpoint base out (§2.6). |
|
|
659
|
+
| `docs` | string | no | Markdown (§2, `docs`). |
|
|
660
|
+
| `wait` | int, string | no | Wait this many milliseconds instead of sending a request (below). |
|
|
661
|
+
|
|
662
|
+
- A step with two method keys is an error, and so is a step with none, unless it is a
|
|
663
|
+
use step, reads a connection or waits.
|
|
664
|
+
- **The URL is authoritative, query string included.** There is no separate block of
|
|
665
|
+
query parameters. Editors show a parameter table as a view over the URL.
|
|
666
|
+
- A step may instead run reusable requests with `use:` (§2.5). A use step has no
|
|
667
|
+
method key.
|
|
668
|
+
- A step may also read a connection, an event stream an earlier step keeps open, with
|
|
669
|
+
`connection:` and no method key (§2.11).
|
|
670
|
+
- A step may only wait, with `wait:` and no method key (below).
|
|
671
|
+
- **Any other key is an error**, so a misspelled key such as `heders:` fails at once
|
|
672
|
+
rather than being ignored. A method key in lower case (`get:`) is reported as such.
|
|
673
|
+
|
|
674
|
+
**`forEach`**, on a setup or teardown step (§2.10), sends the request once for each item
|
|
675
|
+
of a list, each reported as its own result, `teardown › remove grant (item 2 of 3)`:
|
|
676
|
+
|
|
677
|
+
```yaml
|
|
678
|
+
teardown:
|
|
679
|
+
- name: remove grant
|
|
680
|
+
DELETE: '{{baseUrl}}/grants/{{item}}'
|
|
681
|
+
forEach: '{{grantedRoots}}' # a variable holding ["r1", "r2"], or a list written in place
|
|
682
|
+
```
|
|
683
|
+
|
|
684
|
+
- It is for a list of things to prepare or clean up, such as every grant the rows made.
|
|
685
|
+
A step in `steps` cannot have it: a test sends each of its requests once, and a data
|
|
686
|
+
file (§2.8) repeats the steps for each row. Nor can a reusable requests file's steps.
|
|
687
|
+
- Its `{{variables}}` resolve when the step runs, so an earlier step can make the list:
|
|
688
|
+
`ata.set('grantedRoots', ids)` in setup, or with `{ scope: 'run' }` in a row (§2.10).
|
|
689
|
+
The result must be a JSON array.
|
|
690
|
+
- Each item is `{{item}}` in the request and `item` in its scripts. An item that is an
|
|
691
|
+
object or a list is its JSON in the request.
|
|
692
|
+
- An empty list skips the step. Anything that is not a list fails it before anything
|
|
693
|
+
is sent.
|
|
694
|
+
- A use step cannot have `forEach`.
|
|
695
|
+
|
|
696
|
+
**`wait`** makes a step that sends nothing: it waits, then the next step runs. It is for
|
|
697
|
+
something the API does after it answers, such as a search index catching up:
|
|
698
|
+
|
|
699
|
+
```yaml
|
|
700
|
+
steps:
|
|
701
|
+
- name: create order
|
|
702
|
+
POST: '{{baseUrl}}/orders'
|
|
703
|
+
- wait: 2000 # milliseconds; reports call it "wait 2 s"
|
|
704
|
+
- name: order is searchable
|
|
705
|
+
GET: '{{baseUrl}}/search?q={{orderId}}'
|
|
706
|
+
```
|
|
707
|
+
|
|
708
|
+
- It is a whole number of milliseconds, from 0 to 600000 (10 minutes). A fixed wait any
|
|
709
|
+
longer is nearly always waiting for something to be ready, which a step that checks
|
|
710
|
+
for it does better.
|
|
711
|
+
- It may be a string of `{{variables}}`, resolved when the step runs, such as
|
|
712
|
+
`wait: '{{retryAfterMs}}'` with a value an earlier step's `tests` set with `ata.set`.
|
|
713
|
+
What it comes to must be a whole number from 0 to 600000, or the step fails. A string
|
|
714
|
+
that holds no `{{variable}}`, such as `2s`, is an error.
|
|
715
|
+
- A wait step has `wait`, and may have `name`, `docs`, `flags`, `tags` and `exclude` (§2.4,
|
|
716
|
+
§2.9).
|
|
717
|
+
It has no method key, `use`, `connection`, `headers`, `body`, `settings`, `before`,
|
|
718
|
+
`tests`, `forEach`, `useTests` or `base`. Its name defaults to how long it waits:
|
|
719
|
+
`wait 2 s`, `wait 500 ms`, `wait {{retryAfterMs}}`.
|
|
720
|
+
- No script runs for it, the collection's included: there is no request or response.
|
|
721
|
+
Its feature flags, `ata.skipRest`, `bail`, tags and `exclude` apply as to any step.
|
|
722
|
+
- Its result has the method `WAIT`, no response and no checks, and passes once it has
|
|
723
|
+
waited. A cancel, or `timeoutCollection`, stops it at once, as a failure; teardown
|
|
724
|
+
then runs as §2.10 says.
|
|
725
|
+
- It can be a step of `steps`, `setup` and `teardown`, and of a reusable requests file
|
|
726
|
+
(§2.5). An endpoints file holds none (§2.6).
|
|
727
|
+
|
|
728
|
+
### 2.2 `body`
|
|
729
|
+
|
|
730
|
+
A body declares **exactly one** of these keys. Omit `body` for no body.
|
|
731
|
+
|
|
732
|
+
```text
|
|
733
|
+
body: { json: '{ "a": 1 }' } # Content-Type: application/json
|
|
734
|
+
body: { xml: '<a/>' } # application/xml
|
|
735
|
+
body: { text: hello } # text/plain
|
|
736
|
+
body: { form: { a: '1', b: two } } # application/x-www-form-urlencoded
|
|
737
|
+
body: { multipart: { … } } # multipart/form-data (below)
|
|
738
|
+
body: { graphql: { query: '…', variables: { … } } } # application/json
|
|
739
|
+
body: { file: files/order.json } # the file as it is, typed by its extension
|
|
740
|
+
```
|
|
741
|
+
|
|
742
|
+
| Key | Type | Sent as |
|
|
743
|
+
| ----------- | ------------------------------------ | ---------------------------------------------------------- |
|
|
744
|
+
| `json` | string | The text, as written. It must be a string, not a YAML map. |
|
|
745
|
+
| `xml` | string | The text. |
|
|
746
|
+
| `text` | string | The text. |
|
|
747
|
+
| `form` | map of strings | URL-encoded, names and values resolved first. |
|
|
748
|
+
| `multipart` | map of fields (below) | `multipart/form-data`. |
|
|
749
|
+
| `graphql` | `{ query: string, variables?: map }` | `{"query": …, "variables": …}` as JSON. |
|
|
750
|
+
| `file` | string (relative path) | The file's bytes. |
|
|
751
|
+
|
|
752
|
+
- The implied `Content-Type` is added only when `headers` does not declare one.
|
|
753
|
+
- `{{variables}}` resolve in every kind of body. A form's values are resolved and then
|
|
754
|
+
encoded, so a value may hold `&` or `=`.
|
|
755
|
+
- A number in `form` or `multipart` must be quoted: `count: '3'`.
|
|
756
|
+
|
|
757
|
+
Write JSON as a block, so it needs no escaping:
|
|
758
|
+
|
|
759
|
+
```yaml
|
|
760
|
+
body:
|
|
761
|
+
json: |
|
|
762
|
+
{
|
|
763
|
+
"user": "{{username}}",
|
|
764
|
+
"amount": 1200
|
|
765
|
+
}
|
|
766
|
+
```
|
|
767
|
+
|
|
768
|
+
**`multipart`** is keyed by field name, in order:
|
|
769
|
+
|
|
770
|
+
```yaml
|
|
771
|
+
body:
|
|
772
|
+
multipart:
|
|
773
|
+
description: A photo of {{name}} # text
|
|
774
|
+
metadata: # text with a Content-Type of its own
|
|
775
|
+
value: '{"album": "{{album}}"}'
|
|
776
|
+
contentType: application/json
|
|
777
|
+
avatar: # a file from the project folder
|
|
778
|
+
file: files/avatar.png
|
|
779
|
+
contentType: image/png # optional: else from the extension, else application/octet-stream
|
|
780
|
+
filename: me.png # optional: else the file's own name
|
|
781
|
+
tags: [red, blue] # a list sends the name once for each
|
|
782
|
+
```
|
|
783
|
+
|
|
784
|
+
| A field is | Sent as |
|
|
785
|
+
| ----------------------------------- | ------------------------------------------------ |
|
|
786
|
+
| a string | A text part. |
|
|
787
|
+
| `{ value, contentType? }` | A text part with its own `Content-Type`. |
|
|
788
|
+
| `{ file, contentType?, filename? }` | A file part. |
|
|
789
|
+
| a list of the above | One part for each item, all with the field name. |
|
|
790
|
+
|
|
791
|
+
`filename: ''` sends an empty name, as a browser does when no file is chosen. Pair it
|
|
792
|
+
with an empty file for the whole of that request.
|
|
793
|
+
|
|
794
|
+
The `Content-Type` is `multipart/form-data` with a boundary chosen for the request. A
|
|
795
|
+
declared multipart type without a boundary, such as `multipart/mixed`, gets one added.
|
|
796
|
+
A declared boundary is used as written.
|
|
797
|
+
|
|
798
|
+
**Files**, in `body.file` and a multipart part's `file`, are read from the folder of
|
|
799
|
+
the project the step belongs to. For a step of a reusable requests file, that is the
|
|
800
|
+
file's own project, which may be a global one. The path is the same wherever the
|
|
801
|
+
collection sits inside `collections/`.
|
|
802
|
+
|
|
803
|
+
A file that is not there is looked for in the global project (§1.1), as a reusable
|
|
804
|
+
requests file or a base collection is, so projects can share one copy of a file. A
|
|
805
|
+
project's own file of the same path wins. `global:` before the path reads only the
|
|
806
|
+
global project's:
|
|
807
|
+
|
|
808
|
+
```yaml
|
|
809
|
+
body:
|
|
810
|
+
multipart:
|
|
811
|
+
avatar: { file: files/test-png.png } # the project's, else the global project's
|
|
812
|
+
terms: { file: global:files/terms.pdf } # the global project's only
|
|
813
|
+
```
|
|
814
|
+
|
|
815
|
+
- It must be a relative path, written with `/`. An absolute path is refused.
|
|
816
|
+
- A path out of the project folder, such as `../other/files/a.png`, is read from where
|
|
817
|
+
it points, with no global project to fall back to. A `global:` path stays inside the
|
|
818
|
+
global project folder, and is an error in a project that uses none.
|
|
819
|
+
- `{{variables}}` resolve in the path, so a data file row (§2.8) can choose the file.
|
|
820
|
+
- A file is sent byte for byte. `{{…}}` inside it is not a variable.
|
|
821
|
+
- A file that cannot be read stops the step before anything is sent, in its own `body`
|
|
822
|
+
phase, naming the file and each place it was looked for.
|
|
823
|
+
- Where a request is shown (results, reports, `req.body`), a file's bytes appear as
|
|
824
|
+
`‹file files/avatar.png, 1234 bytes›`, and a global project's file as
|
|
825
|
+
`‹file global:files/terms.pdf, 5254 bytes›`, however the step named it.
|
|
826
|
+
|
|
827
|
+
### 2.3 `settings` and `headers`
|
|
828
|
+
|
|
829
|
+
**`settings`** on a collection apply to every step; a step's own settings merge over
|
|
830
|
+
them. Any other key is an error.
|
|
831
|
+
|
|
832
|
+
| Key | Type | Default | Meaning |
|
|
833
|
+
| ----------------- | -------------------- | ------- | ------------------------------------------------------------------------------------------------------ |
|
|
834
|
+
| `timeout` | number ≥ 0 | `0` | Milliseconds for the whole request; `0` means no limit. For an event stream, until its headers arrive. |
|
|
835
|
+
| `followRedirects` | boolean | `true` | Follow 3xx responses. |
|
|
836
|
+
| `maxRedirects` | integer ≥ 0 | `5` | The most redirects followed. |
|
|
837
|
+
| `encodeUrl` | boolean | `true` | Percent-encode what a hand-typed URL left raw, before sending. |
|
|
838
|
+
| `httpVersion` | `auto`, `1.1` or `2` | `auto` | Which HTTP version to send with; see below. |
|
|
839
|
+
| `maxEvents` | integer ≥ 0 | `0` | Stop reading an event stream after this many events; `0` means no limit. |
|
|
840
|
+
| `streamTimeout` | number ≥ 0 | `0` | Stop reading an event stream this many milliseconds after its headers; `0` means no limit. |
|
|
841
|
+
| `untilEvent` | string | none | Stop reading an event stream after the first event of this name, its `event:` line. |
|
|
842
|
+
| `untilTestsPass` | boolean | `true` | Stop reading an event stream once the step's tests pass, tried after each event; see below. |
|
|
843
|
+
|
|
844
|
+
**HTTP versions.** `httpVersion` takes `2` and `1.1` written plain or quoted.
|
|
845
|
+
|
|
846
|
+
- `auto` uses HTTP/2 when an https server picks it, as most do, and HTTP/1.1 otherwise.
|
|
847
|
+
An http:// URL gets HTTP/1.1.
|
|
848
|
+
- `1.1` never offers HTTP/2.
|
|
849
|
+
- `2` is HTTP/2 or an error. An https server that answers with HTTP/1.1 fails the step
|
|
850
|
+
with `NO_HTTP2`. To an http:// URL, HTTP/2 is spoken in the clear from the start (h2c),
|
|
851
|
+
which a server without it cannot follow.
|
|
852
|
+
- HTTP/3 is not supported.
|
|
853
|
+
|
|
854
|
+
Every result says which HTTP version the request went out on, and whether it went on a
|
|
855
|
+
new connection or on one kept open from an earlier request. The same holds for a request
|
|
856
|
+
that fails once its headers are sent, along with how long it ran before it failed. A
|
|
857
|
+
request dropped on a reused connection points at a server or proxy that closes idle
|
|
858
|
+
connections sooner than the client expects.
|
|
859
|
+
|
|
860
|
+
**Event streams.** A response whose `Content-Type` is `text/event-stream` (Server-Sent
|
|
861
|
+
Events) is read as it arrives, whatever the method, and checked as a list of events
|
|
862
|
+
(§3). Nothing in the step marks it as a stream. Reading stops at whichever comes first:
|
|
863
|
+
|
|
864
|
+
- the server closes the stream
|
|
865
|
+
- `maxEvents` events have arrived
|
|
866
|
+
- `streamTimeout` milliseconds have passed since its headers arrived
|
|
867
|
+
- the first event named `untilEvent` has arrived
|
|
868
|
+
- the step's tests pass on the events so far, with `untilTestsPass` (below)
|
|
869
|
+
- 1,000 events or 10 MB, a limit no setting lifts
|
|
870
|
+
|
|
871
|
+
```yaml
|
|
872
|
+
- name: price stream
|
|
873
|
+
GET: '{{baseUrl}}/prices/stream?symbol=ACME'
|
|
874
|
+
headers:
|
|
875
|
+
Accept: text/event-stream
|
|
876
|
+
settings:
|
|
877
|
+
maxEvents: 3
|
|
878
|
+
streamTimeout: 5000
|
|
879
|
+
tests: |
|
|
880
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
881
|
+
ata.expectResponseBodyToHaveProperty('[0].event', 'subscribed')
|
|
882
|
+
ata.expectResponseBodyToHaveProperty('[1].data.price', 100)
|
|
883
|
+
```
|
|
884
|
+
|
|
885
|
+
- Each way of stopping is a normal end, not an error, and the checks run on the events
|
|
886
|
+
that arrived. A check that wanted more fails with the path it missed.
|
|
887
|
+
- Stopped at a number of events, the body ends with the last one.
|
|
888
|
+
- Cancel, or a connection that drops mid-stream, is an error, as for any request.
|
|
889
|
+
- `maxEvents`, `streamTimeout`, `untilEvent` and `untilTestsPass` do nothing to any
|
|
890
|
+
other response.
|
|
891
|
+
- In the desktop app, the events show as they arrive, and Stop ends the reading as
|
|
892
|
+
these do, so the checks run on what came.
|
|
893
|
+
- A step that keeps its stream open for later steps is a connection (§2.11).
|
|
894
|
+
|
|
895
|
+
**Waiting on the tests.** With `untilTestsPass`, on unless set to `false`, the step's
|
|
896
|
+
tests are tried after each event on the events so far, and reading stops at the first
|
|
897
|
+
event they all pass on. That is how to wait for one event among many of the same name,
|
|
898
|
+
picked by what it holds:
|
|
899
|
+
|
|
900
|
+
```yaml
|
|
901
|
+
- name: job finishes
|
|
902
|
+
GET: '{{baseUrl}}/jobs/{{jobId}}/events'
|
|
903
|
+
settings:
|
|
904
|
+
streamTimeout: 30000 # fail if it has not finished in 30 s
|
|
905
|
+
tests: |
|
|
906
|
+
ata.test('done', () => assert.equal(res.body.at(-1).data.state, 'done'))
|
|
907
|
+
```
|
|
908
|
+
|
|
909
|
+
- Reading stops on the event that made the tests pass, so that event is the last one,
|
|
910
|
+
`res.body.at(-1)`.
|
|
911
|
+
- The tests pass when they make at least one check and every check passes. Tests that
|
|
912
|
+
never look at the events are not waited on, as they would pass the same on every
|
|
913
|
+
event: only a body check, strict validation, or reading `res.body`, `res.text` or
|
|
914
|
+
`res.stream` looks at them. A step whose tests check only the status is read as if
|
|
915
|
+
`untilTestsPass` were off.
|
|
916
|
+
- A try changes nothing. What it sets, captures or logs is dropped, and so is an
|
|
917
|
+
`ata.skipRest`. Once the reading stops, the tests run as usual on the events it kept,
|
|
918
|
+
and only that run counts.
|
|
919
|
+
- Anything else that stops the reading first ends it as usual, and the tests fail on
|
|
920
|
+
what came. Set `streamTimeout`, or a stream that stays open waits for ever for an event
|
|
921
|
+
that never comes.
|
|
922
|
+
- Every try runs the tests in full, so a slow `ata.test` slows each event.
|
|
923
|
+
- Turn it off for a check that must see the whole stream, such as no `error` event in
|
|
924
|
+
5 seconds. With it on, such a check passes on the first event, and the reading stops
|
|
925
|
+
there.
|
|
926
|
+
|
|
927
|
+
**Resuming.** A stream is never reconnected: the server closing it ends the step. To
|
|
928
|
+
test that a stream resumes, capture the last event's id and send it as `Last-Event-ID`
|
|
929
|
+
in a later step:
|
|
930
|
+
|
|
931
|
+
```yaml
|
|
932
|
+
- name: first part
|
|
933
|
+
GET: '{{baseUrl}}/feed'
|
|
934
|
+
tests: |
|
|
935
|
+
ata.set('lastId', res.body.at(-1).id)
|
|
936
|
+
- name: the rest
|
|
937
|
+
GET: '{{baseUrl}}/feed'
|
|
938
|
+
headers:
|
|
939
|
+
Last-Event-ID: '{{lastId}}'
|
|
940
|
+
tests: |
|
|
941
|
+
ata.expectResponseBodyToHaveProperty('[0].id', '4')
|
|
942
|
+
```
|
|
943
|
+
|
|
944
|
+
**`headers`** is a map from header name to one of:
|
|
945
|
+
|
|
946
|
+
```yaml
|
|
947
|
+
headers:
|
|
948
|
+
Accept: application/json # a value
|
|
949
|
+
X-Forwarded-For: [10.0.0.1, 10.0.0.2] # a list: sent once per value
|
|
950
|
+
X-Debug: { value: '1', enabled: false, description: turn on to trace } # the long form
|
|
951
|
+
X-Route: [blue, { value: green, enabled: false }] # a list's values can take the long form
|
|
952
|
+
```
|
|
953
|
+
|
|
954
|
+
- **A header sent more than once is a list**, sent once per value in order. A YAML map
|
|
955
|
+
cannot hold the same key twice, so writing the header twice is an error that says to
|
|
956
|
+
use a list. Names are compared ignoring case, so the desktop app saves rows named
|
|
957
|
+
`X-Tag` and `x-tag` as one list, under the spelling written first.
|
|
958
|
+
- A collection's headers are sent with every step. A step header of the same name,
|
|
959
|
+
compared case-insensitively as HTTP does, replaces the collection's for that step.
|
|
960
|
+
- A header with `enabled: false` is not sent. On a step it replaces nothing, so
|
|
961
|
+
switching it off brings back the collection's header. In a list, a value with
|
|
962
|
+
`enabled: false` is left out and the others are sent. A list with every value
|
|
963
|
+
switched off is not sent, and on a step it replaces nothing.
|
|
964
|
+
|
|
965
|
+
### 2.4 `tags`, `stepTags` and `exclude`
|
|
966
|
+
|
|
967
|
+
Tags pick what to run. **A collection's `tags` select the whole collection**: every
|
|
968
|
+
step, in order, sharing one variable scope.
|
|
969
|
+
|
|
970
|
+
```yaml
|
|
971
|
+
id: checkout
|
|
972
|
+
tags: [api, payments] # a run for api or payments runs every step here
|
|
973
|
+
```
|
|
974
|
+
|
|
975
|
+
Some collections have steps that stand alone. There, `stepTags: true` lets each step
|
|
976
|
+
carry its own tags, so a run can pick single steps:
|
|
977
|
+
|
|
978
|
+
```yaml
|
|
979
|
+
id: status-codes
|
|
980
|
+
stepTags: true
|
|
981
|
+
steps:
|
|
982
|
+
- name: not found
|
|
983
|
+
GET: '{{baseUrl}}/status/404'
|
|
984
|
+
tags: [smoke, errors]
|
|
985
|
+
```
|
|
986
|
+
|
|
987
|
+
- **A step must not have `tags` unless its collection has `stepTags: true`.** Most
|
|
988
|
+
collections are a flow, where later steps use what earlier ones set, and running part
|
|
989
|
+
of one fails for reasons that have nothing to do with the API.
|
|
990
|
+
- A tag is letters, digits and `- _ . :` with no spaces (`^[A-Za-z0-9._:-]+$`). Tags are
|
|
991
|
+
case-sensitive and have no prefix. A tag is for grouping only: a feature flag is not
|
|
992
|
+
a tag (§2.9).
|
|
993
|
+
|
|
994
|
+
**Selecting** with `tags`: a collection whose own tags match runs whole. Otherwise, with
|
|
995
|
+
`stepTags: true`, its steps whose tags match run, in order. Otherwise nothing in it runs.
|
|
996
|
+
|
|
997
|
+
**Leaving out** with `notTags`: a collection whose own tags match runs nothing, whatever
|
|
998
|
+
else selects it. With `stepTags: true`, its steps whose tags match are dropped from what
|
|
999
|
+
was selected.
|
|
1000
|
+
|
|
1001
|
+
**`exclude: true`** leaves a collection out of group runs: `ata all`, with or without
|
|
1002
|
+
tags, and a folder named to `ata`. Named on its own, it still runs. Use it for work in
|
|
1003
|
+
progress, a manual-only collection, or one waiting on a fix. `ata` lists what it left
|
|
1004
|
+
out, so a suite never shrinks without saying so.
|
|
1005
|
+
|
|
1006
|
+
**`exclude: true` on a step** leaves just that step out of group runs. Its collection
|
|
1007
|
+
runs without it, and named on its own runs every step:
|
|
1008
|
+
|
|
1009
|
+
```yaml
|
|
1010
|
+
id: orders
|
|
1011
|
+
steps:
|
|
1012
|
+
- name: create order
|
|
1013
|
+
POST: '{{baseUrl}}/orders'
|
|
1014
|
+
- name: refund
|
|
1015
|
+
POST: '{{baseUrl}}/orders/{{orderId}}/refund'
|
|
1016
|
+
exclude: true # waiting on a fix; ata orders still runs it
|
|
1017
|
+
```
|
|
1018
|
+
|
|
1019
|
+
- It is only on a step of `steps`. Setup and teardown run whenever their collection
|
|
1020
|
+
does, and a reusable requests file's steps wherever it is used, so neither has it:
|
|
1021
|
+
exclude the collection, or the use step, instead.
|
|
1022
|
+
- A step the tags would not pick is not run either way. `ata` lists each step it left
|
|
1023
|
+
out as `orders › refund`.
|
|
1024
|
+
- A collection whose every step is excluded is left out of a group run, setup and
|
|
1025
|
+
teardown too.
|
|
1026
|
+
- In the desktop app, Run all leaves it out too, and Send still sends it.
|
|
1027
|
+
- Leaving out a step can make later steps fail, such as one that uses a value it sets.
|
|
1028
|
+
|
|
1029
|
+
### 2.5 Reusable requests and `use:`
|
|
1030
|
+
|
|
1031
|
+
**Reusable requests** are one or more requests that any step can run with **`use:`**,
|
|
1032
|
+
passing values with **`with:`**. They are written in a **reusable requests file**: a
|
|
1033
|
+
collection in `requests/`, directly or one folder down, with a `params:` key, the
|
|
1034
|
+
inputs it takes.
|
|
1035
|
+
|
|
1036
|
+
```yaml
|
|
1037
|
+
# requests/login.yml
|
|
1038
|
+
id: login
|
|
1039
|
+
params:
|
|
1040
|
+
username: { required: true, description: Account to log in as }
|
|
1041
|
+
password: { required: true }
|
|
1042
|
+
expectStatus: 200 # a default
|
|
1043
|
+
steps:
|
|
1044
|
+
- name: log in
|
|
1045
|
+
POST: '{{baseUrl}}/login'
|
|
1046
|
+
body:
|
|
1047
|
+
json: '{ "user": "{{params.username}}", "password": "{{params.password}}" }'
|
|
1048
|
+
tests: |
|
|
1049
|
+
ata.expectResponseStatusCodeToBe(params.expectStatus)
|
|
1050
|
+
if (params.expectStatus === 200) {
|
|
1051
|
+
ata.expectResponseBodyToHaveProperty('token', 'authToken', 'setAsCollectionVariable')
|
|
1052
|
+
}
|
|
1053
|
+
```
|
|
1054
|
+
|
|
1055
|
+
```yaml
|
|
1056
|
+
# collections/checkout.yml
|
|
1057
|
+
id: checkout
|
|
1058
|
+
steps:
|
|
1059
|
+
- use: login
|
|
1060
|
+
with:
|
|
1061
|
+
username: '{{adminUser}}'
|
|
1062
|
+
password: '{{adminPassword}}'
|
|
1063
|
+
- use: login
|
|
1064
|
+
name: rejects a bad password
|
|
1065
|
+
with: { username: alice, password: wrong, expectStatus: 401 }
|
|
1066
|
+
tests: |
|
|
1067
|
+
ata.expectResponseBodyToHaveProperty('error', 'invalid credentials')
|
|
1068
|
+
- name: get profile
|
|
1069
|
+
GET: '{{baseUrl}}/me'
|
|
1070
|
+
```
|
|
1071
|
+
|
|
1072
|
+
**Params.** Each param is a plain default (`expectStatus: 200`), or
|
|
1073
|
+
`{ required: true, default, description }`. In a request a param reads as
|
|
1074
|
+
`{{params.name}}`, and in code as `params.name`, a read-only object holding the real
|
|
1075
|
+
values. No variable can take the place of a `params.` name.
|
|
1076
|
+
|
|
1077
|
+
A default may name variables and other params, as in
|
|
1078
|
+
`email: '{{params.accountId}}@example.com'`. Like a `with:` value, it is resolved once
|
|
1079
|
+
for each use, so `accountId: '{{$uuid}}'` is one id wherever the file reads it.
|
|
1080
|
+
|
|
1081
|
+
**A use step** holds only `use`, `with`, `name`, `tags`, `flags`, `docs` and `tests`. A
|
|
1082
|
+
method key, `headers`, `body`, `settings` or `before` on it is an error, and `with`
|
|
1083
|
+
without `use` is an error too.
|
|
1084
|
+
|
|
1085
|
+
- **Finding the file.** `use: login` is `requests/login.yml` in the project, else in its
|
|
1086
|
+
global project. `use: auth/login` is one folder down. `use: global:login` looks
|
|
1087
|
+
only in the global project.
|
|
1088
|
+
- **`with:`** gives plain values; a param left out takes its default. A string may hold
|
|
1089
|
+
`{{variables}}`, resolved as the file's first request starts, just after the
|
|
1090
|
+
collection's `before.script` has run for it: a value that script sets for each step
|
|
1091
|
+
(§4) reaches the file's requests. A missing required value, a name the file does not
|
|
1092
|
+
take, or a value or default that cannot be resolved stops the file's steps before
|
|
1093
|
+
anything is sent.
|
|
1094
|
+
- **Running.** A use step runs each of the file's steps in turn, in the collection's
|
|
1095
|
+
variable scope, so what one sets the next can read, and so can the steps after the use
|
|
1096
|
+
step. Each request is reported as its own result. When the use step has a `name`,
|
|
1097
|
+
reports use it: `sign in` for a file of one step, `sign in › get profile` for a longer
|
|
1098
|
+
one.
|
|
1099
|
+
- **Layers.** Headers and settings: the collection's, under the file's, under each
|
|
1100
|
+
step's own. Scripts run collection, then file, then step: `before.script` before the
|
|
1101
|
+
request and `tests` after. The use step's own `tests` run last, on the response of
|
|
1102
|
+
the file's step marked `useTests: true`, or else its last step that sends a request:
|
|
1103
|
+
a wait has no response. A use step with `tests` cannot run a file of only waits.
|
|
1104
|
+
`params` is the file's alone: the collection's scripts, and a base collection's or an
|
|
1105
|
+
endpoint's (§2.6, §2.7), never see it. In an endpoint's `before.script`, a segment written as
|
|
1106
|
+
`{{params.id}}` reads as written.
|
|
1107
|
+
- **One level.** A reusable requests file must not `use:` another, and has `params`,
|
|
1108
|
+
not `vars`. A file in `requests/` without `params` is not a reusable requests file.
|
|
1109
|
+
|
|
1110
|
+
**Saving under the caller's name.** A reusable requests file can take the name to save
|
|
1111
|
+
a value under as a param, save it with `ata.set(params.saveAs, …)`, and read it back in
|
|
1112
|
+
its later steps with `{{@params.saveAs}}` (§4). The caller then reads it by the name it
|
|
1113
|
+
chose:
|
|
1114
|
+
|
|
1115
|
+
```yaml
|
|
1116
|
+
# requests/create-user.yml
|
|
1117
|
+
params:
|
|
1118
|
+
saveTokenAs: { required: true }
|
|
1119
|
+
steps:
|
|
1120
|
+
- name: create token
|
|
1121
|
+
POST: '{{authUrl}}/token'
|
|
1122
|
+
tests: ata.set(params.saveTokenAs, res.body.accessToken)
|
|
1123
|
+
- name: get profile
|
|
1124
|
+
GET: '{{baseUrl}}/profile'
|
|
1125
|
+
headers:
|
|
1126
|
+
Authorization: Bearer {{@params.saveTokenAs}}
|
|
1127
|
+
useTests: true # the use step's tests check this response
|
|
1128
|
+
- name: wait for events
|
|
1129
|
+
GET: '{{baseUrl}}/wait'
|
|
1130
|
+
```
|
|
1131
|
+
|
|
1132
|
+
```yaml
|
|
1133
|
+
# a collection
|
|
1134
|
+
- use: create-user
|
|
1135
|
+
name: user 1
|
|
1136
|
+
with: { saveTokenAs: token1 }
|
|
1137
|
+
tests: ata.expectResponseBodyToHaveProperty('email') # on get profile's response
|
|
1138
|
+
- GET: '{{baseUrl}}/orders'
|
|
1139
|
+
headers: { Authorization: 'Bearer {{token1}}' }
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
Only one step of a reusable requests file may have `useTests`, and no other file's
|
|
1143
|
+
steps.
|
|
1144
|
+
|
|
1145
|
+
### 2.6 Endpoint bases
|
|
1146
|
+
|
|
1147
|
+
What every request to one endpoint gets, wherever the request is written. A file in
|
|
1148
|
+
`endpoints/` (the project's own, or its global project's) is a collection whose steps
|
|
1149
|
+
are **endpoints**: a method and a **path pattern**, with the headers, settings, scripts
|
|
1150
|
+
and checks for every request to it.
|
|
1151
|
+
|
|
1152
|
+
```yaml
|
|
1153
|
+
# endpoints/users.yml
|
|
1154
|
+
id: users
|
|
1155
|
+
headers:
|
|
1156
|
+
X-Api: users # every endpoint in this file
|
|
1157
|
+
steps:
|
|
1158
|
+
- GET: /users/{id}
|
|
1159
|
+
headers:
|
|
1160
|
+
Accept: application/json
|
|
1161
|
+
tests: |
|
|
1162
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
1163
|
+
ata.expectResponseBodyToHaveProperty('id', endpoint.id)
|
|
1164
|
+
- POST: /users
|
|
1165
|
+
tests: |
|
|
1166
|
+
ata.expectResponseStatusCodeToBe(201)
|
|
1167
|
+
```
|
|
1168
|
+
|
|
1169
|
+
- An endpoint's URL must be a path starting with `/`. An endpoints file must not hold a
|
|
1170
|
+
use step or a wait step.
|
|
1171
|
+
- **Matching.** A step's request is under the endpoint with its method and a matching
|
|
1172
|
+
path. The path is read from the URL as written. The host, or a leading
|
|
1173
|
+
`{{variable}}` standing for it, is ignored, and so is the query string, so one base
|
|
1174
|
+
serves every environment.
|
|
1175
|
+
- `{name}` in a pattern stands for one path segment, whether a literal value or a
|
|
1176
|
+
`{{variable}}`. A variable never matches a literal segment of a pattern.
|
|
1177
|
+
- When several endpoints match, the one with the most literal segments wins
|
|
1178
|
+
(`/users/me` over `/users/{id}`), then the one listed first. A project's endpoint
|
|
1179
|
+
replaces its global project's with the same method and path.
|
|
1180
|
+
- **`endpoint`** in scripts holds each `{name}`'s value from the step's URL, resolved:
|
|
1181
|
+
`endpoint.id` is `42` for `{{baseUrl}}/users/{{userId}}` with `userId: 42`.
|
|
1182
|
+
- **What applies**, outermost first: the endpoints file's own `headers`, `settings`,
|
|
1183
|
+
`before` and `tests`, then the endpoint's, then the base collection's (§2.7), the
|
|
1184
|
+
collection's, the reusable requests file's (§2.5) and the step's. Nearer headers and
|
|
1185
|
+
settings win.
|
|
1186
|
+
- **A step's own check replaces the base's check of the same thing**: the status, a
|
|
1187
|
+
header by name, or a body property by path. A negative test only says what it expects,
|
|
1188
|
+
so checking for a 404 replaces the base's 200. Checks of other things stay, and named
|
|
1189
|
+
tests (`ata.test`) are never replaced.
|
|
1190
|
+
- **`base: false`** on a step leaves its endpoint base out altogether.
|
|
1191
|
+
|
|
1192
|
+
### 2.7 `extends`: base collections
|
|
1193
|
+
|
|
1194
|
+
```yaml
|
|
1195
|
+
# bases/authenticated.yml
|
|
1196
|
+
id: authenticated
|
|
1197
|
+
headers:
|
|
1198
|
+
Authorization: Bearer {{token}}
|
|
1199
|
+
before:
|
|
1200
|
+
script: |
|
|
1201
|
+
ata.set('requestId', ata.uuidv7())
|
|
1202
|
+
```
|
|
1203
|
+
|
|
1204
|
+
```yaml
|
|
1205
|
+
# collections/checkout.yml
|
|
1206
|
+
id: checkout
|
|
1207
|
+
extends: authenticated
|
|
1208
|
+
steps:
|
|
1209
|
+
- GET: '{{baseUrl}}/cart'
|
|
1210
|
+
```
|
|
1211
|
+
|
|
1212
|
+
A collection that `extends:` a base collection builds on its `headers`, `settings`,
|
|
1213
|
+
`vars`, `before` and `tests`, with its own on top: nearer headers, settings and
|
|
1214
|
+
variables win, and scripts run the base's first.
|
|
1215
|
+
|
|
1216
|
+
- `extends: name` looks in the project's `bases/`, then its global project's.
|
|
1217
|
+
`extends: global:name` looks only in the global project's.
|
|
1218
|
+
- A base collection must have no `steps` and no `params`, and must not `extends`
|
|
1219
|
+
another.
|
|
1220
|
+
- A base that cannot be used (missing, broken, or breaking these rules) stops every
|
|
1221
|
+
step of the collection before anything is sent, saying why.
|
|
1222
|
+
|
|
1223
|
+
### 2.8 Data files
|
|
1224
|
+
|
|
1225
|
+
A collection can be driven by a **data file**: `<id>.csv` or `<id>.json` beside
|
|
1226
|
+
`<id>.yml`, with exactly the same name. The whole collection runs once per row, and each
|
|
1227
|
+
column of the row is a variable for that run: `{{userId}}`, or `ata.get('userId')`.
|
|
1228
|
+
|
|
1229
|
+
```
|
|
1230
|
+
collections/
|
|
1231
|
+
├── users.yml
|
|
1232
|
+
└── users.csv
|
|
1233
|
+
```
|
|
1234
|
+
|
|
1235
|
+
```csv
|
|
1236
|
+
userId,expectedStatus,iterationLabel
|
|
1237
|
+
1001,200,Happy path
|
|
1238
|
+
9999,404,Unknown user
|
|
1239
|
+
```
|
|
1240
|
+
|
|
1241
|
+
- **One run per row.** Every step runs for row 1, then every step again for row 2, and
|
|
1242
|
+
so on. Each run starts from a fresh variable scope, so nothing one row sets leaks into
|
|
1243
|
+
the next, except a value set to last the run (§2.10). Three steps and two rows report
|
|
1244
|
+
six results.
|
|
1245
|
+
- **Precedence.** A row's values sit over the environment and under what code sets
|
|
1246
|
+
(§4).
|
|
1247
|
+
- **`iterationLabel`**, an optional column, names the row in reports, as in
|
|
1248
|
+
`Iteration 2 (Unknown user) - get user`. It is a variable like the others.
|
|
1249
|
+
- **CSV** follows RFC 4180: a header row, then one row per line. A value in double
|
|
1250
|
+
quotes may hold commas and line breaks, and `""` is a quote. **Every CSV value is a
|
|
1251
|
+
string**: a zip code `01234` stays `01234`. A short row leaves its last columns empty;
|
|
1252
|
+
a row with more values than the header is an error.
|
|
1253
|
+
- **JSON** is an array of objects, one per row. Values keep their type, which must be
|
|
1254
|
+
string, number, boolean or null. A row must not name a column twice: JSON readers keep
|
|
1255
|
+
only the last value, so it is an error rather than a value quietly lost.
|
|
1256
|
+
- **`.csv` wins** when both exist. The name must match exactly, case included.
|
|
1257
|
+
- A data file must be at most 10 MB and have at least one row. The header must not have
|
|
1258
|
+
an empty or repeated column name. A data file that will not read makes the collection
|
|
1259
|
+
broken: `ata` reports it and runs nothing from it.
|
|
1260
|
+
- With `bail`, a failing step also stops the rows still to come.
|
|
1261
|
+
|
|
1262
|
+
In the desktop app, a single step runs with one chosen row. **Run all** runs every row,
|
|
1263
|
+
as `ata` does.
|
|
1264
|
+
|
|
1265
|
+
### 2.9 Feature flags
|
|
1266
|
+
|
|
1267
|
+
A collection or a step says which feature flags it needs, and runs only when they hold:
|
|
1268
|
+
|
|
1269
|
+
```yaml
|
|
1270
|
+
id: checkout
|
|
1271
|
+
flags: { newCheckout: true } # the whole collection runs only when newCheckout is on
|
|
1272
|
+
steps:
|
|
1273
|
+
- name: total (new)
|
|
1274
|
+
GET: '{{baseUrl}}/checkout/total'
|
|
1275
|
+
flags: { newCheckout: true }
|
|
1276
|
+
- name: total (old)
|
|
1277
|
+
GET: '{{baseUrl}}/checkout/total'
|
|
1278
|
+
flags: { newCheckout: false } # only one of the pair ever runs
|
|
1279
|
+
- name: v2 pricing
|
|
1280
|
+
GET: '{{baseUrl}}/prices'
|
|
1281
|
+
flags: { pricingVersion: v2 } # any value, not just on/off
|
|
1282
|
+
```
|
|
1283
|
+
|
|
1284
|
+
- **Every flag named must have the value given.** A collection's `flags` apply to each
|
|
1285
|
+
of its steps, and a use step's to each of its reusable requests. Values compare as
|
|
1286
|
+
text, so `true` matches `true` or `"true"`, and `2` matches `"2"`.
|
|
1287
|
+
- **A step whose flags do not hold is skipped**, not failed. It sends nothing and is
|
|
1288
|
+
reported as skipped with the reason, such as `feature flag newCheckout is off`. A
|
|
1289
|
+
skipped step never fails a run.
|
|
1290
|
+
- **A flag the run does not know is an error.** Otherwise a typo, or a flag deleted from
|
|
1291
|
+
the flag service, would quietly run or skip the wrong tests.
|
|
1292
|
+
- **In code, `ata.flag(name)`** returns a flag's value, for checking something
|
|
1293
|
+
different rather than skipping a step.
|
|
1294
|
+
- A flag name is letters, digits and `- _ .` (`^[A-Za-z0-9_][A-Za-z0-9_.-]*$`). A value
|
|
1295
|
+
is a string, number or boolean.
|
|
1296
|
+
- Skipping an early step can make later steps fail, such as a login that sets a token.
|
|
1297
|
+
Flags on the whole collection are usually the safer choice.
|
|
1298
|
+
|
|
1299
|
+
**Where the values come from.** Each environment file gives its flags (§6):
|
|
1300
|
+
|
|
1301
|
+
```yaml
|
|
1302
|
+
# environments/staging.yml
|
|
1303
|
+
vars: { baseUrl: https://staging.example.com }
|
|
1304
|
+
flags:
|
|
1305
|
+
command: node scripts/flags.mjs staging # prints the flags as JSON
|
|
1306
|
+
values: # used without a command, and for any flag its output leaves out
|
|
1307
|
+
newCheckout: false
|
|
1308
|
+
```
|
|
1309
|
+
|
|
1310
|
+
Lowest precedence first: the global project's environment's `values`, the project's
|
|
1311
|
+
own, what the `command` prints, then overrides.
|
|
1312
|
+
|
|
1313
|
+
- **`command`** runs a program once before any test of a run, in the folder of the
|
|
1314
|
+
project whose environment file names it. It must print, on standard output, a JSON
|
|
1315
|
+
object of flag names to string, number or boolean values, such as
|
|
1316
|
+
`{ "newCheckout": true, "pricingVersion": "v2" }`.
|
|
1317
|
+
- **It runs the same way on every platform, without a shell.** The command is split into
|
|
1318
|
+
words at spaces. The first word is the program, found on the `PATH` or given as a path
|
|
1319
|
+
from the project folder, and the rest are its arguments. `'…'` or `"…"` keeps spaces
|
|
1320
|
+
in a word. Inside `"…"`, `\"` stands for `"` and `\\` for `\`; anywhere else `\` is an
|
|
1321
|
+
ordinary character, so a Windows path works as written. Pipes, `&&`, redirection,
|
|
1322
|
+
`$VAR` and `%VAR%` mean nothing. Put logic in a script and run it with its interpreter,
|
|
1323
|
+
as in `node scripts/flags.mjs staging`. On Windows, a `.cmd` or `.bat` script such as
|
|
1324
|
+
`npx` cannot be started this way, so name the program it runs instead.
|
|
1325
|
+
- It inherits the environment `ata` or the app runs in, so a flag service's API key
|
|
1326
|
+
comes from CI or the shell, never from the repository. The desktop app adds the `PATH`
|
|
1327
|
+
a terminal would have, so `node` is found however the app was opened.
|
|
1328
|
+
- If it exits with anything but `0`, prints anything else, or takes longer than 60
|
|
1329
|
+
seconds, **nothing runs**, and what it wrote to standard error is shown. A command that
|
|
1330
|
+
takes too long is stopped, with anything it started.
|
|
1331
|
+
- A project's command wins over its global project's for the same environment.
|
|
1332
|
+
- **Overrides** win over everything: `ata … --flag newCheckout=false` (any number of
|
|
1333
|
+
them), or the environment variable `ATA_FLAG_newCheckout=false`. The desktop app has an
|
|
1334
|
+
override per flag. `true` and `false` are booleans, a number is a number, and anything
|
|
1335
|
+
else is text.
|
|
1336
|
+
|
|
1337
|
+
Every report records the flag values a run used and where each came from.
|
|
1338
|
+
|
|
1339
|
+
### 2.10 Setup, teardown and values that last the run
|
|
1340
|
+
|
|
1341
|
+
`setup` and `teardown` are lists of steps, like `steps`, run once around it:
|
|
1342
|
+
|
|
1343
|
+
```yaml
|
|
1344
|
+
id: approved-domains
|
|
1345
|
+
setup: # once, before the first row
|
|
1346
|
+
- use: seed-admin # log in, grant an admin role, wait for it to land
|
|
1347
|
+
teardown: # once, after the last row, even when a row failed
|
|
1348
|
+
- name: remove the grant
|
|
1349
|
+
DELETE: '{{baseUrl}}/admins/{{adminId}}'
|
|
1350
|
+
steps: # once per row of approved-domains.csv, as before
|
|
1351
|
+
- name: approve the domains
|
|
1352
|
+
PUT: '{{baseUrl}}/orgs/{{orgId}}/domains'
|
|
1353
|
+
```
|
|
1354
|
+
|
|
1355
|
+
- **Order.** Setup, then the steps once per data row (or once, with no data file), then
|
|
1356
|
+
teardown. They work the same with or without a data file.
|
|
1357
|
+
- **What setup sets lasts the run.** Every value setup sets or captures is there for
|
|
1358
|
+
every row and for teardown. It sits over the environment and under a row's values
|
|
1359
|
+
(§4).
|
|
1360
|
+
- **A row keeps a value for the rows after it** with
|
|
1361
|
+
`ata.set(name, value, { scope: 'run' })`. Anything else a row sets lasts only that
|
|
1362
|
+
row, as §2.8 says.
|
|
1363
|
+
- **Setup stops at its first step that fails**, with `bail` or without: a setup step
|
|
1364
|
+
counts on the ones before it. The setup steps after it and the rows are not run, and
|
|
1365
|
+
count as skipped. Teardown still runs.
|
|
1366
|
+
- **Teardown always runs**, every step of it, after failures and after `bail` — and after
|
|
1367
|
+
a run is cancelled or stopped by `timeoutCollection`, when it has at most 60 seconds of
|
|
1368
|
+
its own. A cancel or timeout before anything ran runs no teardown, since there is
|
|
1369
|
+
nothing to clean up after. One that comes while teardown runs stops it where it is,
|
|
1370
|
+
and so does a second Stop in the desktop app.
|
|
1371
|
+
- **The collection applies to them**: its headers, settings, `before`, `tests` and
|
|
1372
|
+
flags, as to any step. Their steps may be use steps, and only theirs may have
|
|
1373
|
+
`forEach` (§2.1). They carry no `tags`: they run whenever the collection does.
|
|
1374
|
+
- A reusable requests file, a base collection and an endpoints file have no setup or
|
|
1375
|
+
teardown.
|
|
1376
|
+
- Reports name their results `setup › log in` and `teardown › remove the grant`. Running
|
|
1377
|
+
a single step in the desktop app does not run them.
|
|
1378
|
+
|
|
1379
|
+
### 2.11 Connections: a stream across steps
|
|
1380
|
+
|
|
1381
|
+
A request whose response is an event stream (§2.3) can keep it open for later steps, as
|
|
1382
|
+
a **connection** named by `connection:`. A later step with `connection:` and no method
|
|
1383
|
+
key sends nothing: it reads the events the connection holds. The steps between can do
|
|
1384
|
+
what those events are about:
|
|
1385
|
+
|
|
1386
|
+
```yaml
|
|
1387
|
+
steps:
|
|
1388
|
+
- name: watch orders
|
|
1389
|
+
GET: '{{baseUrl}}/orders/events'
|
|
1390
|
+
connection: orders # keep the stream open as "orders"
|
|
1391
|
+
settings:
|
|
1392
|
+
untilEvent: subscribed # read until the server confirms, then go on
|
|
1393
|
+
- name: place order
|
|
1394
|
+
POST: '{{baseUrl}}/orders'
|
|
1395
|
+
body:
|
|
1396
|
+
json: '{ "sku": "ACME-1" }'
|
|
1397
|
+
tests: |
|
|
1398
|
+
ata.expectResponseBodyToHaveProperty('id', 'orderId', 'setAsCollectionVariable')
|
|
1399
|
+
- name: order created
|
|
1400
|
+
connection: orders # no method key: reads the connection
|
|
1401
|
+
settings:
|
|
1402
|
+
streamTimeout: 5000 # read until the tests pass, for up to 5 s
|
|
1403
|
+
tests: |
|
|
1404
|
+
ata.test('the order placed', () => {
|
|
1405
|
+
const last = res.body.at(-1)
|
|
1406
|
+
assert.equal(last.event, 'order.created')
|
|
1407
|
+
assert.equal(last.data.id, ata.get('orderId'))
|
|
1408
|
+
})
|
|
1409
|
+
```
|
|
1410
|
+
|
|
1411
|
+
- **Opening.** The step sends its request and reads the stream as any step does, until
|
|
1412
|
+
its `maxEvents`, `streamTimeout` or `untilEvent`, or its tests pass. Then, rather than
|
|
1413
|
+
closing it, it keeps reading in the background, holding the events that arrive for
|
|
1414
|
+
the next step that reads the connection. With none of the three set, it reads no
|
|
1415
|
+
events and goes straight on: `untilTestsPass` alone does not make it wait.
|
|
1416
|
+
- **Reading.** A reading step takes the events held since the last step read them,
|
|
1417
|
+
then waits for more, until its `maxEvents`, `streamTimeout` or `untilEvent`, its tests
|
|
1418
|
+
pass, or the server closes the stream. With none of the three set, it takes what is
|
|
1419
|
+
held and ends without waiting, as `held`, or sooner, as `testsPassed`, at the event
|
|
1420
|
+
its tests pass on.
|
|
1421
|
+
- **What a step leaves.** The events after the one a step stopped at stay held for the
|
|
1422
|
+
next step that reads the connection.
|
|
1423
|
+
- **What a reading step sees.** Its body is the events it took (§3). Its status and
|
|
1424
|
+
headers are those of the response that opened the connection, `req` is that request,
|
|
1425
|
+
and `res.stream.at` counts from that response's headers.
|
|
1426
|
+
- A reading step may have `name`, `settings`, `before`, `tests`, `tags`, `flags`,
|
|
1427
|
+
`useTests` and `docs`, and nothing that would build a request: no `headers`, `body`,
|
|
1428
|
+
`base` or `forEach`. A step opening a connection has no `forEach`, and a use step has
|
|
1429
|
+
no `connection`.
|
|
1430
|
+
- **A connection lasts the run.** One opened in `setup` is read by every data row; one
|
|
1431
|
+
opened in a row, by that row's later steps. Every connection closes when the run ends,
|
|
1432
|
+
cancelled or not. A step opening a name already open closes the old connection first,
|
|
1433
|
+
whatever its own response is.
|
|
1434
|
+
- A connection the server closes keeps the events it held: the next step reads them,
|
|
1435
|
+
then ends with `close`. A connection holds at most 1,000 events or 10 MB; at that it
|
|
1436
|
+
stops reading, and the step that reads it ends with `limit`.
|
|
1437
|
+
- Reading a connection no step has opened is an error, and the step's scripts do not
|
|
1438
|
+
run.
|
|
1439
|
+
- **In the desktop app**, a connection opened by sending a step stays open for the steps
|
|
1440
|
+
sent after it, until that step is sent again, the connection is closed from above the
|
|
1441
|
+
step list, Run all starts, or the app quits. Run all, like `ata`, opens its own and
|
|
1442
|
+
closes them when it ends.
|
|
1443
|
+
|
|
1444
|
+
---
|
|
1445
|
+
|
|
1446
|
+
## 3. Checking a response
|
|
1447
|
+
|
|
1448
|
+
A step's checks are calls to functions on `ata`, in its `tests` (§5 lists them).
|
|
1449
|
+
[FUNCTIONS.md](./FUNCTIONS.md) documents each one, with examples. This section is the
|
|
1450
|
+
rules they share.
|
|
1451
|
+
|
|
1452
|
+
```yaml
|
|
1453
|
+
tests: |
|
|
1454
|
+
ata.useStrictValidation()
|
|
1455
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
1456
|
+
ata.expectResponseToHaveHeader('Content-Type', /^application\/json/)
|
|
1457
|
+
ata.expectResponseBodyToHaveProperty('user.name', 'Ada')
|
|
1458
|
+
ata.expectResponseBodyToHaveProperty('user.score', 100, 'integerWithin2')
|
|
1459
|
+
ata.expectResponseBodyToHaveProperty('user.token', 'sessionToken', 'setAsCollectionVariable')
|
|
1460
|
+
ata.expectResponseBodyToHaveUnorderedArray('user.roles', ['admin', 'editor'])
|
|
1461
|
+
ata.ignoreResponseBodyProperty('user.lastSeen')
|
|
1462
|
+
```
|
|
1463
|
+
|
|
1464
|
+
### Paths
|
|
1465
|
+
|
|
1466
|
+
A path addresses the body in dot or bracket notation: `user.name`; `groups.0.name` or
|
|
1467
|
+
`groups[0].name` for an index; and `sessions[].id` for a property of every item.
|
|
1468
|
+
|
|
1469
|
+
A key that holds a `.`, `[` or `]`, or is empty, goes in brackets as a JSON string:
|
|
1470
|
+
`jwt.payload["https://example.com/id"]`, `modules[""].edition`. Reports show such keys
|
|
1471
|
+
the same way.
|
|
1472
|
+
|
|
1473
|
+
A path can also be a list of keys: `['jwt', 'payload', 'https://example.com/id']`.
|
|
1474
|
+
Each item is one key, whatever it holds, and a number is its digits, so
|
|
1475
|
+
`['groups', 0, 'name']` reads an index. Every function that takes a path takes a list
|
|
1476
|
+
too, and so does `pathToProperty`.
|
|
1477
|
+
|
|
1478
|
+
A path that runs into a `null` before its end, such as `phoneNumber.number` when
|
|
1479
|
+
`phoneNumber` is `null`, reads as that `null` for a check that the value is `null`. For
|
|
1480
|
+
any other check the property is not present.
|
|
1481
|
+
|
|
1482
|
+
### Body conversion
|
|
1483
|
+
|
|
1484
|
+
A JSON body is used as it is. An XML body is converted: the root element is the single
|
|
1485
|
+
top-level key, namespace prefixes and attributes are dropped, an element holding only
|
|
1486
|
+
text becomes that string (an empty one `""`), and a repeated element becomes an array.
|
|
1487
|
+
A `text/plain` or HTML body is the single property `plaintext`.
|
|
1488
|
+
|
|
1489
|
+
An event stream (§2.3) is a list with one item per event:
|
|
1490
|
+
|
|
1491
|
+
- **`data`** is the event's `data:` lines joined with a line feed: their JSON value when
|
|
1492
|
+
they parse as JSON, so `[1].data.price` is a number, and otherwise the text, such as
|
|
1493
|
+
`[DONE]`.
|
|
1494
|
+
- **`event`** and **`id`** are there only when that event's own lines set them. Neither
|
|
1495
|
+
carries over to the next event.
|
|
1496
|
+
- Comments (`: heartbeat`) and a block with no `data:` are not events, `retry:` is not
|
|
1497
|
+
kept, and an event the stream ended in the middle of is dropped.
|
|
1498
|
+
|
|
1499
|
+
```text
|
|
1500
|
+
event: price
|
|
1501
|
+
id: 41
|
|
1502
|
+
data: {"symbol":"ACME","price":100}
|
|
1503
|
+
|
|
1504
|
+
: heartbeat
|
|
1505
|
+
|
|
1506
|
+
data: [DONE]
|
|
1507
|
+
|
|
1508
|
+
```
|
|
1509
|
+
|
|
1510
|
+
reads as
|
|
1511
|
+
|
|
1512
|
+
```json
|
|
1513
|
+
[{ "event": "price", "id": "41", "data": { "symbol": "ACME", "price": 100 } }, { "data": "[DONE]" }]
|
|
1514
|
+
```
|
|
1515
|
+
|
|
1516
|
+
The path `''` is the whole list, so `expectResponseBodyToHaveUnorderedArray('', …)` finds
|
|
1517
|
+
events whose order is not guaranteed. Under strict validation every event's properties
|
|
1518
|
+
need accounting for, so a small `maxEvents` keeps a busy stream practical.
|
|
1519
|
+
|
|
1520
|
+
### Values and patterns
|
|
1521
|
+
|
|
1522
|
+
- A value compares with its type: `'12345'` does not equal `12345`.
|
|
1523
|
+
- A `RegExp` is a pattern, flags included, tested against the value as text.
|
|
1524
|
+
- Header names match case-insensitively. The status and header values compare as text.
|
|
1525
|
+
|
|
1526
|
+
### `specialHandling`
|
|
1527
|
+
|
|
1528
|
+
The last argument of a check may be one of these strings:
|
|
1529
|
+
|
|
1530
|
+
| String | Means |
|
|
1531
|
+
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
|
1532
|
+
| _(none, and no value)_ | The property or header exists. |
|
|
1533
|
+
| `notThisExpectedKey` | It must not exist. Pass `null` as the value. |
|
|
1534
|
+
| `notThisExpectedValue` | It must exist and not equal the value (or not match the RegExp). |
|
|
1535
|
+
| `setAsCollectionVariable` | Capture it into the variable the value names, for the rest of the run. |
|
|
1536
|
+
| `setAsEnvironmentVariable` | The same. Nothing is written back to the environment file. |
|
|
1537
|
+
| `dateAsEpoch` | Compare by calendar day: a number is a seconds offset from when the step started, a string a date whose first ten characters must match. |
|
|
1538
|
+
| `dateWithin<X>Sec` | A date within X seconds of the value, as in `dateWithin5Sec`. |
|
|
1539
|
+
| `integerWithin<X>` | A number within X of the value, as in `integerWithin2`. |
|
|
1540
|
+
| `isArray` | An array; its contents are not checked. |
|
|
1541
|
+
| `isArrayAndEmpty` / `isArrayAndNotEmpty` | An empty or non-empty array. |
|
|
1542
|
+
| `isArrayAndHasLength` | An array of exactly the value's length. |
|
|
1543
|
+
|
|
1544
|
+
A mistake in a call, such as an unknown `specialHandling`, is reported as a failed check
|
|
1545
|
+
saying so, and the checks after it still run.
|
|
1546
|
+
|
|
1547
|
+
### Unordered arrays
|
|
1548
|
+
|
|
1549
|
+
`ata.expectResponseBodyToHaveUnorderedArray(path, list)` passes when the array holds
|
|
1550
|
+
every item of a simple `list`, in any order. A `RegExp` in the list is a pattern some item
|
|
1551
|
+
must match as text, and so is one held by an object in the list:
|
|
1552
|
+
`[/^admin/, { name: /^Grace/ }]`. A pattern never matches an object or array item.
|
|
1553
|
+
An object in the list matches an item holding each of its properties, and the item may
|
|
1554
|
+
have others; a property that is itself an object is matched the same way, so
|
|
1555
|
+
`{ data: { status: 'reversed' } }` finds an item whose `data` has that `status` among
|
|
1556
|
+
other properties. An array compares whole.
|
|
1557
|
+
|
|
1558
|
+
A list of `{ pathToProperty, expectedValue, specialHandling? }` objects describes **one**
|
|
1559
|
+
item, property by property; call it once per item. A property may appear twice, once to
|
|
1560
|
+
check it and once to capture it:
|
|
1561
|
+
|
|
1562
|
+
```js
|
|
1563
|
+
ata.expectResponseBodyToHaveUnorderedArray('users', [
|
|
1564
|
+
{ pathToProperty: 'name', expectedValue: 'Ada' },
|
|
1565
|
+
{ pathToProperty: 'id', expectedValue: 'adaId', specialHandling: 'setAsCollectionVariable' }
|
|
1566
|
+
])
|
|
1567
|
+
```
|
|
1568
|
+
|
|
1569
|
+
`ata.expectResponseBodyToHaveUnorderedArrayNotThisItem(path, list)` passes when no item
|
|
1570
|
+
matches. A simple list may hold patterns here too, and a `compareValue` may be a
|
|
1571
|
+
`RegExp`.
|
|
1572
|
+
|
|
1573
|
+
- **Each call prefers items an earlier call did not match.** Two calls with the same
|
|
1574
|
+
description find two items when there are two, so each capture and strict validation
|
|
1575
|
+
see a different one. A sort starts this over, since its indexes name other items.
|
|
1576
|
+
- **A list of one `notThisExpectedValue` entry depends on strict validation.** Without
|
|
1577
|
+
it, the entry means no item has that value, so an empty array passes. With it, the
|
|
1578
|
+
entry means one item whose value is something else, as any list does. The step's last
|
|
1579
|
+
word on strict validation decides.
|
|
1580
|
+
|
|
1581
|
+
### Strict validation
|
|
1582
|
+
|
|
1583
|
+
`ata.useStrictValidation()` fails the step unless **every** property of the body is
|
|
1584
|
+
checked, ignored or captured.
|
|
1585
|
+
|
|
1586
|
+
- `null`, `""` and empty arrays or objects never need a check of their own.
|
|
1587
|
+
- A check on a value's content accounts for that value and everything beneath it. A
|
|
1588
|
+
shape-only check (`isArray`, `isArrayAndHasLength`, or an object's existence) does not
|
|
1589
|
+
vouch for what is inside.
|
|
1590
|
+
- It is judged after the collection's and the step's `tests` have both run, over what
|
|
1591
|
+
either checked.
|
|
1592
|
+
- A binary or HTML body has no properties to leave unchecked, so strict validation
|
|
1593
|
+
passes. A check on one still fails.
|
|
1594
|
+
|
|
1595
|
+
### Sorting
|
|
1596
|
+
|
|
1597
|
+
`ata.sortResponseBodyArrays(property)` sorts every array of objects holding the
|
|
1598
|
+
property, before the checks after it.
|
|
1599
|
+
|
|
1600
|
+
- The property may be a path (`id.value`), and nested arrays are sorted too.
|
|
1601
|
+
- Items without the property go last.
|
|
1602
|
+
- Values compare alphanumerically, so `Group 2` comes before `Group 10`.
|
|
1603
|
+
- Indexed paths refer to the sorted order.
|
|
1604
|
+
|
|
1605
|
+
---
|
|
1606
|
+
|
|
1607
|
+
## 4. Variables
|
|
1608
|
+
|
|
1609
|
+
Variables come from `vars` in `project.yml` and in collections, the chosen environment,
|
|
1610
|
+
a data file's row, and what code sets while a step runs.
|
|
1611
|
+
|
|
1612
|
+
```yaml
|
|
1613
|
+
vars:
|
|
1614
|
+
apiVersion: '2' # plain values only: string, number, boolean or null
|
|
1615
|
+
|
|
1616
|
+
before:
|
|
1617
|
+
script: |
|
|
1618
|
+
ata.set('traceId', ata.uuidv7())
|
|
1619
|
+
ata.set('today', ata.date('%Y-%m-%d', 0, 'utc'))
|
|
1620
|
+
```
|
|
1621
|
+
|
|
1622
|
+
**`vars` hold plain data.** A value must be a string, number, boolean or null. Anything
|
|
1623
|
+
computed, such as an id, a date, a random number or a value built from other
|
|
1624
|
+
variables, is set in `before.script` with `ata.set` (§5). A collection's
|
|
1625
|
+
`before.script` runs before every step, so a value set there is fresh for each.
|
|
1626
|
+
|
|
1627
|
+
### Interpolation
|
|
1628
|
+
|
|
1629
|
+
`{{name}}` resolves in the URL, headers and body, and recursively in what it resolves
|
|
1630
|
+
to. Whitespace inside the braces is ignored. In code, read a variable with
|
|
1631
|
+
`ata.get(name)` instead.
|
|
1632
|
+
|
|
1633
|
+
- **Values keep their type.** A string that is exactly one reference returns the
|
|
1634
|
+
variable's own value, so a variable written as `true` arrives as a boolean. Anything
|
|
1635
|
+
else is text, and `null` becomes the empty string inside it, except in a `json` body,
|
|
1636
|
+
where it is written `null`, so `"website": {{site}}` stays valid JSON.
|
|
1637
|
+
- **An unknown variable is an error**, never literal text. The step fails in its own
|
|
1638
|
+
`interpolate` phase, naming the variable, and nothing is sent. A reference loop, or a
|
|
1639
|
+
chain more than 16 deep, fails the same way.
|
|
1640
|
+
- **`{{@name}}` reads the variable `name` names.** With `saveAs: token1`,
|
|
1641
|
+
`{{@saveAs}}` is the value of `token1`. The name may itself be built from variables.
|
|
1642
|
+
A name that is not text, or names no variable, fails like an unknown variable. A
|
|
1643
|
+
reusable requests file uses it to read what it saved under its caller's name (§2.5).
|
|
1644
|
+
- **In YAML, quote a value that starts with `{{`.** Unquoted, YAML reads `{` as a map.
|
|
1645
|
+
|
|
1646
|
+
### Built-in variables
|
|
1647
|
+
|
|
1648
|
+
These are usable anywhere a variable is, with no declaration:
|
|
1649
|
+
|
|
1650
|
+
| Reference | Value |
|
|
1651
|
+
| ------------------- | -------------------------------------- |
|
|
1652
|
+
| `{{$uuid}}` | A random UUID, new for each reference. |
|
|
1653
|
+
| `{{$timestamp}}` | Epoch milliseconds. |
|
|
1654
|
+
| `{{$isoTimestamp}}` | An ISO-8601 instant. |
|
|
1655
|
+
| `{{$randomInt}}` | A whole number from 0 to 999. |
|
|
1656
|
+
|
|
1657
|
+
### Resolution order
|
|
1658
|
+
|
|
1659
|
+
Lowest precedence first:
|
|
1660
|
+
|
|
1661
|
+
```
|
|
1662
|
+
global project vars → project vars → base collection vars → collection vars
|
|
1663
|
+
→ environment → what lasts the run → data file row → ata.set and captures
|
|
1664
|
+
→ process environment
|
|
1665
|
+
```
|
|
1666
|
+
|
|
1667
|
+
The environment is the global project's file of that name, if there is one, with the
|
|
1668
|
+
project's own over it.
|
|
1669
|
+
|
|
1670
|
+
**The process environment only overrides a name that another layer already declares.**
|
|
1671
|
+
Otherwise `PATH`, `HOME` and every credential on the machine would be reachable from a
|
|
1672
|
+
request URL. To let CI override a value, declare the variable, usually in an environment
|
|
1673
|
+
file and often as a secret (§6).
|
|
1674
|
+
|
|
1675
|
+
A name is matched exactly, case included, on every platform. Windows ignores case in its
|
|
1676
|
+
environment, but a variable called `path` or `username` is still not replaced by the
|
|
1677
|
+
system's `Path` or `USERNAME`.
|
|
1678
|
+
|
|
1679
|
+
---
|
|
1680
|
+
|
|
1681
|
+
## 5. Code: `tests` and `before.script`
|
|
1682
|
+
|
|
1683
|
+
A step, or a collection for every step, carries JavaScript. `before.script` prepares the
|
|
1684
|
+
request, and `tests` checks the response:
|
|
1685
|
+
|
|
1686
|
+
```yaml
|
|
1687
|
+
- name: get user
|
|
1688
|
+
GET: '{{baseUrl}}/users/7'
|
|
1689
|
+
before:
|
|
1690
|
+
script: |
|
|
1691
|
+
ata.set('traceId', ata.uuidv7())
|
|
1692
|
+
tests: |
|
|
1693
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
1694
|
+
ata.expectResponseBodyToHaveProperty('user.name', 'Ada')
|
|
1695
|
+
ata.expectResponseBodyToHaveProperty('user.nickname', null, 'notThisExpectedKey')
|
|
1696
|
+
|
|
1697
|
+
const ids = res.body.user.accounts.map((a) => a.id)
|
|
1698
|
+
ata.test('account ids are unique', () => assert.equal(new Set(ids).size, ids.length))
|
|
1699
|
+
```
|
|
1700
|
+
|
|
1701
|
+
- `before` must hold only `script`.
|
|
1702
|
+
- Write code as a block scalar (`|`) so it needs no quoting.
|
|
1703
|
+
- **Order:** the collection's `before.script`, the step's `before.script`, the request,
|
|
1704
|
+
then the collection's `tests` and the step's `tests`. Strict validation is judged last.
|
|
1705
|
+
|
|
1706
|
+
### The `ata` object
|
|
1707
|
+
|
|
1708
|
+
There is nothing to import or load. [FUNCTIONS.md](./FUNCTIONS.md) documents each
|
|
1709
|
+
function, with examples.
|
|
1710
|
+
|
|
1711
|
+
| Function | In `before.script` |
|
|
1712
|
+
| ------------------------------------------------------------------------------------------------ | :----------------: |
|
|
1713
|
+
| `ata.expectResponseStatusCodeToBe(expected, specialHandling?)` | |
|
|
1714
|
+
| `ata.expectResponseToHaveHeader(name, expected?, specialHandling?)` | |
|
|
1715
|
+
| `ata.expectResponseToHaveHeaderCount(name, expectedCount)` | |
|
|
1716
|
+
| `ata.expectResponseBodyToHaveProperty(path, expected?, specialHandling?)` | |
|
|
1717
|
+
| `ata.expectResponseBodyToHaveUnorderedArray(path, list)` | |
|
|
1718
|
+
| `ata.expectResponseBodyToHaveUnorderedArrayNotThisItem(path, list)` | |
|
|
1719
|
+
| `ata.ignoreResponseBodyProperty(path)` | |
|
|
1720
|
+
| `ata.ignoreResponseBodyArrayObjectProperty(arrayPath, propertyPath)` | |
|
|
1721
|
+
| `ata.sortResponseBodyArrays(property)` | |
|
|
1722
|
+
| `ata.useStrictValidation(enabled = true)` | |
|
|
1723
|
+
| `ata.test(name, fn)`: a named check that passes unless `fn` throws or rejects; `fn` may be async | |
|
|
1724
|
+
| `ata.get(name)`: a variable's current value | ✓ |
|
|
1725
|
+
| `ata.set(name, value)`: set a variable for the rest of the run | ✓ |
|
|
1726
|
+
| `ata.set(name, value, { scope: 'run' })`: the same, lasting past this data row too (§2.10) | ✓ |
|
|
1727
|
+
| `ata.skip(reason?)`: send nothing for this step, and report it skipped with the reason | ✓ |
|
|
1728
|
+
| `ata.skipRest(reason?)`: skip the rest of this row's steps; in `before.script`, this one too | ✓ |
|
|
1729
|
+
| `ata.flag(name)`: a feature flag's value; an unknown flag is an error | ✓ |
|
|
1730
|
+
| `ata.uuid()`: a random (version 4) UUID | ✓ |
|
|
1731
|
+
| `ata.uuidv7()`: a time-ordered (version 7) UUID | ✓ |
|
|
1732
|
+
| `ata.randomInt(min, max)`: a whole number, both ends included | ✓ |
|
|
1733
|
+
| `ata.date(format, secondsOffset = 0, timeZone = 'local')` | ✓ |
|
|
1734
|
+
|
|
1735
|
+
`ata.set` keeps a string, number, boolean or null as it is, and stores an object or array
|
|
1736
|
+
as JSON text. `undefined` is stored as `null`, so a variable set to nothing still reads
|
|
1737
|
+
`== null`, and `{{name}}` resolves as a null does (§4).
|
|
1738
|
+
|
|
1739
|
+
Calling a response check in `before.script` is an error, and so is `ata.skip` in
|
|
1740
|
+
`tests`, where the request has been sent. After `ata.skip` the script runs to its end,
|
|
1741
|
+
and no later `before.script` runs. `ata.skipRest` skips steps of this row only: the next
|
|
1742
|
+
row, and teardown, still run.
|
|
1743
|
+
|
|
1744
|
+
`ata.date` formats with strftime specifiers: `%Y %y %m %d %e %H %I %M %S %L %p %b %B %a
|
|
1745
|
+
%A %j %Z %z %s %F %T %%`. An unrecognized specifier is left in the output, so a typo is
|
|
1746
|
+
visible. `timeZone` is `local`, `utc`, an IANA name such as `America/New_York`, or a
|
|
1747
|
+
military zone letter (`U` is -08:00, not UTC).
|
|
1748
|
+
|
|
1749
|
+
### Other globals
|
|
1750
|
+
|
|
1751
|
+
| Global | What it is |
|
|
1752
|
+
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
1753
|
+
| `res` | `tests` only: `status`, `statusText`, `headers` (lower-cased names, a repeated header's values joined with `, `), `header(name)`, `headerValues(name)` (every value, as a list), `body` (parsed JSON, converted XML, an event stream's events, or text), `text`, `time` (ms), `size` (bytes), and for an event stream `stream` (below). |
|
|
1754
|
+
| `req` | `method`, `url`, `headers`, `body`: as sent in `tests`, as written in `before.script`, where a script may change `headers` and `body` (below). |
|
|
1755
|
+
| `assert` | Node's strict `assert`, for use inside `ata.test`. |
|
|
1756
|
+
| `console` | Captured into the step's result. |
|
|
1757
|
+
| `params` | A reusable requests file's params (§2.5), in its own scripts and in the tests of the use step running it. |
|
|
1758
|
+
| `endpoint` | An endpoint base's `{name}` values (§2.6), in its scripts and in every script of a step under it. |
|
|
1759
|
+
| `checks` | The project's check files (below). |
|
|
1760
|
+
|
|
1761
|
+
**`res` for an event stream** (§2.3): `res.body` is its list of events, as checks see it
|
|
1762
|
+
(§3), `res.text` the stream as received, and `res.time` runs until the reading stopped.
|
|
1763
|
+
`res.stream` has the rest, and is absent for any other response:
|
|
1764
|
+
|
|
1765
|
+
- `endedBy`: what stopped the reading: `close`, `maxEvents`, `streamTimeout`,
|
|
1766
|
+
`untilEvent`, `testsPassed` (`untilTestsPass`), `limit`, `stopped` (the desktop app's
|
|
1767
|
+
Stop button), or `held` (a step on a connection that waited for nothing, §2.11)
|
|
1768
|
+
- `at`: for each event, the milliseconds from the response headers to its arrival
|
|
1769
|
+
- `connection`: for a step that opens or reads a connection, `{ name, open }`: its name,
|
|
1770
|
+
and whether it was still open when the step ended
|
|
1771
|
+
|
|
1772
|
+
```js
|
|
1773
|
+
ata.test('first price within 2s', () => assert.ok(res.stream.at[0] < 2000))
|
|
1774
|
+
```
|
|
1775
|
+
|
|
1776
|
+
Also available are the language itself and the web-standard globals: timers, `URL`,
|
|
1777
|
+
`URLSearchParams`, `TextEncoder`, `TextDecoder`, `atob`, `btoa`, `structuredClone` and
|
|
1778
|
+
`crypto`.
|
|
1779
|
+
|
|
1780
|
+
**There is no `require`, `import`, `process`, filesystem or `fetch`.** A collection must
|
|
1781
|
+
run the same on any machine, and a request belongs in a step, where it is recorded.
|
|
1782
|
+
|
|
1783
|
+
- A script that throws stops that script. It is reported with its line, and the step is
|
|
1784
|
+
marked as errored. Checks made before it are kept.
|
|
1785
|
+
- A script that runs for more than 10 seconds is stopped.
|
|
1786
|
+
|
|
1787
|
+
**Changing the request in `before.script`.** `req` there is the request as written,
|
|
1788
|
+
`{{variables}}` still in it. A script may change it before it is sent, and a later
|
|
1789
|
+
`before.script` sees the change:
|
|
1790
|
+
|
|
1791
|
+
```js
|
|
1792
|
+
const claims = JSON.parse(req.body)
|
|
1793
|
+
if (!params.crmContactId) delete claims.crm_contact_id // leave the member out
|
|
1794
|
+
req.body = JSON.stringify(claims)
|
|
1795
|
+
req.headers['X-Trace'] = '{{traceId}}'
|
|
1796
|
+
delete req.headers['X-Debug']
|
|
1797
|
+
```
|
|
1798
|
+
|
|
1799
|
+
- `req.body` is text, and can be changed for a `json`, `xml`, `text` or `graphql` body.
|
|
1800
|
+
A form, multipart or file body is built from its parts, so it cannot.
|
|
1801
|
+
- `req.headers` is a map of names to values: add, change or delete them. A header sent
|
|
1802
|
+
more than once reads as its values joined with `, `, as `res.headers` does, and is
|
|
1803
|
+
still sent once per value unless the script changes it. Set an array to send one
|
|
1804
|
+
header per value.
|
|
1805
|
+
- Variables in what the script writes resolve afterwards, as in the file.
|
|
1806
|
+
- Anything else is a pre-request error, and nothing is sent.
|
|
1807
|
+
|
|
1808
|
+
### Check files
|
|
1809
|
+
|
|
1810
|
+
`checks/*.js` in a project, and in its global project, hold functions every script can
|
|
1811
|
+
call as `checks.<file>.<function>`:
|
|
1812
|
+
|
|
1813
|
+
```js
|
|
1814
|
+
// checks/pagination.js
|
|
1815
|
+
export function expectPage({ size }) {
|
|
1816
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
1817
|
+
ata.expectResponseBodyToHaveProperty('page.size', size)
|
|
1818
|
+
}
|
|
1819
|
+
```
|
|
1820
|
+
|
|
1821
|
+
```js
|
|
1822
|
+
// in a step's tests
|
|
1823
|
+
checks.pagination.expectPage({ size: 20 })
|
|
1824
|
+
```
|
|
1825
|
+
|
|
1826
|
+
- A function uses the calling script's `ata`, `res`, `req`, `assert` and `params`, so
|
|
1827
|
+
what it checks is reported on the step that called it.
|
|
1828
|
+
- A file exports with `export function name`, `export async function name` or
|
|
1829
|
+
`export const name =`.
|
|
1830
|
+
- The file name, without `.js`, must be a JavaScript identifier
|
|
1831
|
+
(`^[A-Za-z_$][\w$]*$`). A file with any other name is not loaded.
|
|
1832
|
+
- A project's file replaces its global project's file of the same name.
|
|
1833
|
+
|
|
1834
|
+
---
|
|
1835
|
+
|
|
1836
|
+
## 6. `environments/<name>.yml`
|
|
1837
|
+
|
|
1838
|
+
```yaml
|
|
1839
|
+
name: staging # optional: else the file name
|
|
1840
|
+
vars:
|
|
1841
|
+
baseUrl: https://staging.example.com
|
|
1842
|
+
strictValidation: true # a real boolean
|
|
1843
|
+
apiKey: { secret: true } # the value comes from the process environment or .env
|
|
1844
|
+
region: { value: eu, description: Where the test accounts live }
|
|
1845
|
+
flags: # feature flags for this environment (§2.9)
|
|
1846
|
+
command: node scripts/flags.mjs staging
|
|
1847
|
+
values: { newCheckout: true }
|
|
1848
|
+
```
|
|
1849
|
+
|
|
1850
|
+
Any other top-level key is an error.
|
|
1851
|
+
|
|
1852
|
+
| Key | Type | Meaning |
|
|
1853
|
+
| ------- | ------ | ------------------------------------------------------------------------------- |
|
|
1854
|
+
| `name` | string | The environment's name. Defaults to the file name without `.yml`. |
|
|
1855
|
+
| `vars` | map | Each value is a plain value, or `{ value?, secret?: true, description? }`. |
|
|
1856
|
+
| `flags` | map | `command` (string) and `values` (flag name → string, number or boolean) (§2.9). |
|
|
1857
|
+
|
|
1858
|
+
**Secrets.** A variable written `{ secret: true }` never has its value in a file or a
|
|
1859
|
+
report. Its value is read from the process environment variable of **exactly the same
|
|
1860
|
+
name**, case included, else from `.env` at the project's root, else from `.env` at its
|
|
1861
|
+
global project's root.
|
|
1862
|
+
|
|
1863
|
+
- A secret with no value anywhere fails the run, naming it. An empty credential is never
|
|
1864
|
+
sent.
|
|
1865
|
+
- Reports replace a secret's value with `[secret: NAME]`, wherever it appears. So does
|
|
1866
|
+
the desktop app where it shows a request as sent, or what a script wrote with
|
|
1867
|
+
`console`.
|
|
1868
|
+
- `.env` holds `NAME=value` lines. `#` starts a comment line, `export ` before a name is
|
|
1869
|
+
allowed, and a value may be quoted. It must not be committed.
|
|
1870
|
+
|
|
1871
|
+
An environment is selected by its `name`, else its file name. A project can use its own
|
|
1872
|
+
`environments/` and its global project's (§1.1). Two files of the same name are one
|
|
1873
|
+
environment, with the project's values over the global project's.
|
|
1874
|
+
|
|
1875
|
+
---
|
|
1876
|
+
|
|
1877
|
+
## 7. Editing files
|
|
1878
|
+
|
|
1879
|
+
Files are meant to be written by hand, by tools and by agents, and kept in git.
|
|
1880
|
+
|
|
1881
|
+
- **Accelerator edits in place.** Saving a file Accelerator did not change leaves it as it was,
|
|
1882
|
+
byte for byte. Changing one field changes that field. Comments, key order, quoting and
|
|
1883
|
+
formatting survive, and so does every step not touched.
|
|
1884
|
+
- **An edited file is written with LF line endings.**
|
|
1885
|
+
- **Accelerator writes only against what it read.** If a file changed on disk while it was
|
|
1886
|
+
being edited, nothing is written, and the change is shown instead.
|
|
1887
|
+
- Code (`tests`, `before.script`) that Accelerator writes is a `|` block, even for one line.
|
|
1888
|
+
|
|
1889
|
+
When writing files yourself, especially from a program or an agent:
|
|
1890
|
+
|
|
1891
|
+
- Use two-space indentation and block style. Flow style (`[a, b]`, `{ secret: true }`)
|
|
1892
|
+
is fine for short values.
|
|
1893
|
+
- Quote any string that starts with `{`, `[`, `*`, `&`, `!`, `%`, `@` or `` ` ``, or
|
|
1894
|
+
that holds `: ` or ` #`.
|
|
1895
|
+
- Write JSON bodies and code as `|` blocks.
|
|
1896
|
+
- Keep `id` equal to the file name, and change both together.
|
|
1897
|
+
- Check the result against [Appendix A](#appendix-a-validation-rules).
|
|
1898
|
+
|
|
1899
|
+
---
|
|
1900
|
+
|
|
1901
|
+
## Appendix A. Validation rules
|
|
1902
|
+
|
|
1903
|
+
A file that breaks one of these rules is reported with a message naming the rule. A
|
|
1904
|
+
broken collection is shown as broken, and `ata` reports it as failed without running
|
|
1905
|
+
it. Rules checked at run time fail the step, or the run, before anything is sent.
|
|
1906
|
+
|
|
1907
|
+
**Every document**
|
|
1908
|
+
|
|
1909
|
+
- It must be YAML that parses, holding a mapping, in a file ending in `.yml`.
|
|
1910
|
+
- Unknown keys are errors in: a collection's top level, a step, `settings`, `before`,
|
|
1911
|
+
`body`, `project.yml`, `tls`, an environment file, an environment's `flags`, a param
|
|
1912
|
+
spec, `settings.yml` and `rules.yml`.
|
|
1913
|
+
|
|
1914
|
+
**Collections**
|
|
1915
|
+
|
|
1916
|
+
- `id` is present, equals the file name without `.yml`, and matches
|
|
1917
|
+
`^[A-Za-z0-9][A-Za-z0-9._-]*$`.
|
|
1918
|
+
- `id` is unique in its home (`collections/`, `requests/`, `bases/` or `endpoints/`),
|
|
1919
|
+
ignoring case.
|
|
1920
|
+
- There is no `name` key (use `id`).
|
|
1921
|
+
- A step has `tags` only if the collection has `stepTags: true`.
|
|
1922
|
+
- With `params`: no step is a use step or has `exclude`, and there is no `vars`, `setup`
|
|
1923
|
+
or `teardown`.
|
|
1924
|
+
- A `setup` or `teardown` step has no `tags` or `exclude`.
|
|
1925
|
+
- A data file, when present, reads (§2.8).
|
|
1926
|
+
|
|
1927
|
+
**Steps**
|
|
1928
|
+
|
|
1929
|
+
- A step has exactly one method key, out of `GET`, `POST`, `PUT`, `PATCH`, `DELETE`,
|
|
1930
|
+
`HEAD` and `OPTIONS`, in capitals, and its value is a string. A use step, a step
|
|
1931
|
+
reading a connection and a wait step have none.
|
|
1932
|
+
- A step has no key outside the table in §2.1.
|
|
1933
|
+
- A use step has no method key, `headers`, `body`, `settings` or `before`. `with` is used
|
|
1934
|
+
only with `use`.
|
|
1935
|
+
- `before` holds only `script`. `before.set` is not supported.
|
|
1936
|
+
- There is no `expect:` key; checks go in `tests`.
|
|
1937
|
+
- `base` is only ever `false`.
|
|
1938
|
+
- `forEach` is a string, only on a setup or teardown step, and a use step has none.
|
|
1939
|
+
- `useTests` is only ever `true`, only on a reusable requests file's step, and on one
|
|
1940
|
+
step at most.
|
|
1941
|
+
- A step reading a connection has no `headers`, `body`, `base` or `forEach`. A step
|
|
1942
|
+
opening one has no `forEach`, and a use step has no `connection` (§2.11).
|
|
1943
|
+
- A wait step's `wait` is a whole number from 0 to 600000, or a string holding a
|
|
1944
|
+
`{{variable}}`, and it holds nothing but `wait`, `name`, `docs`, `flags`, `tags` and
|
|
1945
|
+
`exclude`.
|
|
1946
|
+
At run time its `{{variables}}` must come to a whole number from 0 to 600000 (§2.1).
|
|
1947
|
+
|
|
1948
|
+
**Values**
|
|
1949
|
+
|
|
1950
|
+
- A variable value, in `vars`, `with` or a param, is a string, number, boolean or null.
|
|
1951
|
+
- A header is a string, `{ value, enabled?, description? }`, or a list of either.
|
|
1952
|
+
- A tag matches `^[A-Za-z0-9._:-]+$`.
|
|
1953
|
+
- A flag name matches `^[A-Za-z0-9_][A-Za-z0-9_.-]*$`, and a flag value is a string,
|
|
1954
|
+
number or boolean.
|
|
1955
|
+
- A connection name matches `^[A-Za-z0-9_][A-Za-z0-9_.-]*$`.
|
|
1956
|
+
- A `settings` value has its type in §2.3.
|
|
1957
|
+
|
|
1958
|
+
**Bodies**
|
|
1959
|
+
|
|
1960
|
+
- A body declares exactly one of `json`, `xml`, `text`, `form`, `multipart`, `graphql`
|
|
1961
|
+
and `file`.
|
|
1962
|
+
- `json`, `xml` and `text` are strings, and `form` is a map of strings.
|
|
1963
|
+
- A multipart field is a string, `{ value, contentType? }`,
|
|
1964
|
+
`{ file, contentType?, filename? }`, or a non-empty list of them.
|
|
1965
|
+
- `file` paths are relative.
|
|
1966
|
+
|
|
1967
|
+
**Library files**
|
|
1968
|
+
|
|
1969
|
+
- A reusable requests file (`requests/`) has `params`, uses no other, and has no
|
|
1970
|
+
`vars`.
|
|
1971
|
+
- A base collection (`bases/`) has no `steps`, `setup`, `teardown` or `params`, and no
|
|
1972
|
+
`extends`.
|
|
1973
|
+
- An endpoint (`endpoints/`) has a URL that is a path starting with `/`, and no use
|
|
1974
|
+
steps, connections or `exclude`. An endpoints file has no `setup` or `teardown`.
|
|
1975
|
+
- A check file's name is a JavaScript identifier; if it isn't, the file is not loaded.
|
|
1976
|
+
- `collections/`, `requests/`, `endpoints/` and `bases/` hold files at most one folder
|
|
1977
|
+
down.
|
|
1978
|
+
|
|
1979
|
+
**Portability (§1.2)**
|
|
1980
|
+
|
|
1981
|
+
- A name a file refers to is spelled as it is on disk, case included: a `use:` or
|
|
1982
|
+
`extends:` name, `uses`, a `tls.ca` file and a file a body sends.
|
|
1983
|
+
- The folders and files ata looks for in a project are lower case: `collections/`, not
|
|
1984
|
+
`Collections/`.
|
|
1985
|
+
- No two names in `collections/`, `requests/`, `endpoints/`, `bases/`, `environments/`
|
|
1986
|
+
or `checks/` differ only in case.
|
|
1987
|
+
- No file or folder name is one Windows refuses.
|
|
1988
|
+
|
|
1989
|
+
**Projects**
|
|
1990
|
+
|
|
1991
|
+
- `uses` and `tls.ca` entries are relative paths.
|
|
1992
|
+
- The folder `uses` names has a `project.yml`, and does not itself `uses` another.
|
|
1993
|
+
- Each `tls.ca` file exists and holds a PEM or DER certificate.
|
|
1994
|
+
- `settings.yml` exists for `ata`; a global project's is optional. The `environmentType`
|
|
1995
|
+
a run ends up with, if any, names an environment that exists.
|
|
1996
|
+
- In `rules.yml`, each rule has a value of the type §1.4 gives it, a pattern compiles,
|
|
1997
|
+
`tests.only` lists `ata`, and `guide` is a string. What breaks a rule is a finding of `ata lint`, never an
|
|
1998
|
+
error: it stops nothing (§1.4).
|
|
1999
|
+
|
|
2000
|
+
**At run time**
|
|
2001
|
+
|
|
2002
|
+
- Every `{{variable}}` resolves, with no loop and no chain deeper than 16.
|
|
2003
|
+
- Every secret has a value.
|
|
2004
|
+
- Every feature flag named is known to the run.
|
|
2005
|
+
- The flag `command`, if any, has every quote closed, starts, and exits `0` within 60
|
|
2006
|
+
seconds, printing a JSON object.
|
|
2007
|
+
- Every file a body names can be read from the project folder, or its global
|
|
2008
|
+
project's (§2.2).
|
|
2009
|
+
- Every `use:` and `extends:` names a usable file, and every `with:` suits its file.
|
|
2010
|
+
`ata get` checks these, and the body files named without `{{variables}}`, without
|
|
2011
|
+
running anything (§1.3).
|
|
2012
|
+
- A step's `forEach` resolves to a JSON array.
|
|
2013
|
+
- A step reading a connection finds it open: a step before it in the run opened it.
|
|
2014
|
+
- In `{{@name}}`, `name` holds text naming a variable that exists.
|
|
2015
|
+
- A `before.script` sets `req.body` to text, and only for a `json`, `xml`, `text` or
|
|
2016
|
+
`graphql` body; `req.headers` stays a map.
|
|
2017
|
+
- `ata.set`'s `scope`, when given, is `'run'`.
|
|
2018
|
+
|
|
2019
|
+
---
|
|
2020
|
+
|
|
2021
|
+
## Appendix B. A complete project
|
|
2022
|
+
|
|
2023
|
+
```
|
|
2024
|
+
shop/
|
|
2025
|
+
├── project.yml
|
|
2026
|
+
├── settings.yml
|
|
2027
|
+
├── collections/
|
|
2028
|
+
│ ├── health.yml
|
|
2029
|
+
│ └── users/
|
|
2030
|
+
│ ├── profile.yml
|
|
2031
|
+
│ ├── profile.csv
|
|
2032
|
+
│ └── avatar.yml
|
|
2033
|
+
├── requests/
|
|
2034
|
+
│ └── login.yml
|
|
2035
|
+
├── checks/
|
|
2036
|
+
│ └── common.js
|
|
2037
|
+
├── files/
|
|
2038
|
+
│ └── avatar.png
|
|
2039
|
+
├── environments/
|
|
2040
|
+
│ └── staging.yml
|
|
2041
|
+
└── .env apiKey=… (not committed)
|
|
2042
|
+
```
|
|
2043
|
+
|
|
2044
|
+
```yaml
|
|
2045
|
+
# project.yml
|
|
2046
|
+
name: Shop
|
|
2047
|
+
vars:
|
|
2048
|
+
apiVersion: '2'
|
|
2049
|
+
```
|
|
2050
|
+
|
|
2051
|
+
```yaml
|
|
2052
|
+
# settings.yml
|
|
2053
|
+
environmentType: staging
|
|
2054
|
+
limitConcurrency: 2
|
|
2055
|
+
generateJUnitResults: true
|
|
2056
|
+
```
|
|
2057
|
+
|
|
2058
|
+
```yaml
|
|
2059
|
+
# environments/staging.yml
|
|
2060
|
+
vars:
|
|
2061
|
+
baseUrl: https://staging.example.com
|
|
2062
|
+
adminUser: admin@example.com
|
|
2063
|
+
apiKey: { secret: true }
|
|
2064
|
+
flags:
|
|
2065
|
+
values: { avatars: true }
|
|
2066
|
+
```
|
|
2067
|
+
|
|
2068
|
+
```yaml
|
|
2069
|
+
# requests/login.yml
|
|
2070
|
+
id: login
|
|
2071
|
+
params:
|
|
2072
|
+
username: { required: true }
|
|
2073
|
+
steps:
|
|
2074
|
+
- name: log in
|
|
2075
|
+
POST: '{{baseUrl}}/v{{apiVersion}}/login'
|
|
2076
|
+
headers:
|
|
2077
|
+
X-Api-Key: '{{apiKey}}'
|
|
2078
|
+
body:
|
|
2079
|
+
json: '{ "user": "{{params.username}}" }'
|
|
2080
|
+
tests: |
|
|
2081
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
2082
|
+
ata.expectResponseBodyToHaveProperty('token', 'token', 'setAsCollectionVariable')
|
|
2083
|
+
```
|
|
2084
|
+
|
|
2085
|
+
```js
|
|
2086
|
+
// checks/common.js
|
|
2087
|
+
export function expectJson() {
|
|
2088
|
+
ata.expectResponseToHaveHeader('Content-Type', /^application\/json/)
|
|
2089
|
+
}
|
|
2090
|
+
```
|
|
2091
|
+
|
|
2092
|
+
```yaml
|
|
2093
|
+
# collections/health.yml
|
|
2094
|
+
id: health
|
|
2095
|
+
tags: [smoke]
|
|
2096
|
+
steps:
|
|
2097
|
+
- name: service is up
|
|
2098
|
+
GET: '{{baseUrl}}/health'
|
|
2099
|
+
tests: |
|
|
2100
|
+
ata.expectResponseStatusCodeToBe(200)
|
|
2101
|
+
checks.common.expectJson()
|
|
2102
|
+
```
|
|
2103
|
+
|
|
2104
|
+
```yaml
|
|
2105
|
+
# collections/users/profile.yml
|
|
2106
|
+
id: profile
|
|
2107
|
+
steps:
|
|
2108
|
+
- use: login # sets token for the steps after it
|
|
2109
|
+
with: { username: '{{adminUser}}' }
|
|
2110
|
+
|
|
2111
|
+
- name: get user
|
|
2112
|
+
GET: '{{baseUrl}}/v{{apiVersion}}/users/{{userId}}'
|
|
2113
|
+
headers:
|
|
2114
|
+
Authorization: Bearer {{token}}
|
|
2115
|
+
tests: |
|
|
2116
|
+
ata.expectResponseStatusCodeToBe(ata.get('expectedStatus'))
|
|
2117
|
+
```
|
|
2118
|
+
|
|
2119
|
+
```csv
|
|
2120
|
+
userId,expectedStatus,iterationLabel
|
|
2121
|
+
1001,200,Known user
|
|
2122
|
+
9999,404,Unknown user
|
|
2123
|
+
```
|
|
2124
|
+
|
|
2125
|
+
```yaml
|
|
2126
|
+
# collections/users/avatar.yml
|
|
2127
|
+
id: avatar
|
|
2128
|
+
flags: { avatars: true } # runs only where the environment turns avatars on
|
|
2129
|
+
steps:
|
|
2130
|
+
- use: login
|
|
2131
|
+
with: { username: '{{adminUser}}' }
|
|
2132
|
+
|
|
2133
|
+
- name: upload avatar
|
|
2134
|
+
POST: '{{baseUrl}}/v{{apiVersion}}/users/1001/avatar'
|
|
2135
|
+
headers:
|
|
2136
|
+
Authorization: Bearer {{token}}
|
|
2137
|
+
body:
|
|
2138
|
+
multipart:
|
|
2139
|
+
caption: Profile photo
|
|
2140
|
+
image: { file: files/avatar.png }
|
|
2141
|
+
tests: |
|
|
2142
|
+
ata.expectResponseStatusCodeToBe(201)
|
|
2143
|
+
```
|
|
2144
|
+
|
|
2145
|
+
`ata all` runs `health` and `avatar` once each, and `profile` twice: once for each row
|
|
2146
|
+
of `profile.csv`.
|