@schwabyio/gta 0.9.0 → 0.10.0

package/README.md

@@ -108,6 +108,8 @@ The steps within a collection always run in order.
 
 The exit code is `0` when everything passed, `1` when something failed, and `2` when
 `gta` could not run at all, such as with a bad setting or an unknown collection.
+`gta get` exits `1` when a collection is broken, or has a `use:` or `extends:` a run
+would stop at.
 
 ## Settings
 

package/dist/SPEC.md

@@ -269,6 +269,10 @@ comma-separated: `--tags smoke,api`.
 | `gta all`             | Every collection, except those with `exclude: true` (§2.4). |
 | `gta smoke,checkout/` | The collections and directories named, in that order.       |
 
+`gta get` also reports each `use:` and `extends:` that would stop a run, in every
+collection, including those `gta all` leaves out (Appendix A). It exits `1` when it finds
+one, or a broken collection, and `0` otherwise.
+
 A collection is named by its `id`, or by its place (`checkout/sessions`). A directory is
 named by its name, and `checkout/` names only the directory. `--flag name=value` sets a
 feature flag (§2.9), and `--json` prints the results as JSON. The exit code is `0` when
@@ -320,6 +324,8 @@ Any key not listed here is an error.
 | ---------- | ------------------- | -------- | ------- | ------------------------------------------------------------------------------ |
 | `id`       | string              | **yes**  |         | The file name without `.yml` (below).                                          |
 | `steps`    | list of steps       | no       | `[]`    | The requests, in run order (§2.1).                                             |
+| `setup`    | list of steps       | no       |         | Run once before `steps`; what it sets lasts the run (§2.10).                   |
+| `teardown` | list of steps       | no       |         | Run once after the rest, even when a step failed (§2.10).                      |
 | `docs`     | string              | no       |         | Markdown.                                                                      |
 | `tags`     | list of tags        | no       |         | Tags that select the whole collection (§2.4).                                  |
 | `stepTags` | boolean             | no       | `false` | `true` lets steps carry their own tags (§2.4).                                 |
@@ -376,6 +382,8 @@ Methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`.
 | `tests`    | string       | no       | The checks, as JavaScript: calls on `gta` (§3) and any other code (§5).   |
 | `tags`     | list of tags | no       | The step's own tags. Only with `stepTags: true` on the collection (§2.4). |
 | `flags`    | map          | no       | Feature flags the step needs (§2.9).                                      |
+| `forEach`  | string       | no       | Send the request once for each item of a list (below).                    |
+| `useTests` | `true`       | no       | In a request set: the use step's `tests` check this response (§2.5).      |
 | `base`     | `false`      | no       | `false` leaves the step's endpoint base out (§2.6).                       |
 | `docs`     | string       | no       | Markdown.                                                                 |
 
@@ -387,6 +395,23 @@ Methods: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`.
 - **Any other key is an error**, so a misspelled key such as `heders:` fails at once
   rather than being ignored. A method key in lower case (`get:`) is reported as such.
 
+**`forEach`** sends the request once for each item of a list, each reported as its own
+result, `remove grant (item 2 of 3)`:
+
+```yaml
+- name: remove grant
+  DELETE: '{{baseUrl}}/grants/{{item}}'
+  forEach: '{{grantedRoots}}' # a variable holding ["r1", "r2"], or a list written in place
+```
+
+- Its `{{variables}}` resolve when the step runs, so an earlier step can make the list,
+  as `gta.set('grantedRoots', ids)` does. The result must be a JSON array.
+- Each item is `{{item}}` in the request and `item` in its scripts. An item that is an
+  object or a list is its JSON in the request.
+- An empty list skips the step. Anything that is not a list fails it before anything
+  is sent.
+- A use step cannot have `forEach`.
+
 ### 2.2 `body`
 
 A body declares **exactly one** of these keys. Omit `body` for no body.
@@ -450,6 +475,9 @@ body:
 | `{ file, contentType?, filename? }` | A file part.                                     |
 | a list of the above                 | One part for each item, all with the field name. |
 
+`filename: ''` sends an empty name, as a browser does when no file is chosen. Pair it
+with an empty file for the whole of that request.
+
 The `Content-Type` is `multipart/form-data` with a boundary chosen for the request. A
 declared multipart type without a boundary, such as `multipart/mixed`, gets one added.
 A declared boundary is used as written.
@@ -581,6 +609,10 @@ steps:
 `{{params.name}}`, and in code as `params.name`, a read-only object holding the real
 values. No variable can take the place of a `params.` name.
 
+A default may name variables and other params, as in
+`email: '{{params.accountId}}@example.com'`. Like a `with:` value, it is resolved once,
+when the set starts, so `accountId: '{{$uuid}}'` is one id wherever the set reads it.
+
 **A use step** holds only `use`, `with`, `name`, `tags`, `flags`, `docs` and `tests`. A
 method key, `headers`, `body`, `settings` or `before` on it is an error, and `with`
 without `use` is an error too.
@@ -589,18 +621,54 @@ without `use` is an error too.
   global project. `use: auth/login` is one directory down. `use: global:login` looks
   only in the global project.
 - **`with:`** gives plain values; a param left out takes its default. A string may hold
-  `{{variables}}`, resolved when the set starts. A missing required value, or a name the
-  set does not take, stops the step before anything is sent.
+  `{{variables}}`, resolved when the set starts. A missing required value, a name the
+  set does not take, or a value or default that cannot be resolved stops the set's
+  steps before anything is sent.
 - **Running.** A use step runs each of the set's steps in turn, in the collection's
   variable scope, so what one sets the next can read, and so can the steps after the use
-  step. Each request is reported as its own result.
+  step. Each request is reported as its own result. When the use step has a `name`,
+  reports use it: `sign in` for a set of one step, `sign in › get profile` for a longer
+  one.
 - **Layers.** Headers and settings: the collection's, under the set's, under each
   step's own. Scripts run collection, then set, then step: `before.script` before the
-  request and `tests` after. The use step's own `tests` run last, after the set's last
-  step.
+  request and `tests` after. The use step's own `tests` run last, on the response of
+  the set's step marked `useTests: true`, or else its last step.
 - **One level.** A request set must not `use:` another, and has `params`, not `vars`. A
   file in `requests/` without `params` is not a request set.
 
+**Saving under the caller's name.** A set can take the name to save a value under as a
+param, save it with `gta.set(params.saveAs, …)`, and read it back in its later steps
+with `{{@params.saveAs}}` (§4). The caller then reads it by the name it chose:
+
+```yaml
+# requests/create-user.yml
+params:
+  saveTokenAs: { required: true }
+steps:
+  - name: create token
+    POST: '{{authUrl}}/token'
+    tests: gta.set(params.saveTokenAs, res.body.accessToken)
+  - name: get profile
+    GET: '{{baseUrl}}/profile'
+    headers:
+      Authorization: Bearer {{@params.saveTokenAs}}
+    useTests: true # the use step's tests check this response
+  - name: wait for events
+    GET: '{{baseUrl}}/wait'
+```
+
+```yaml
+# a collection
+- use: create-user
+  name: user 1
+  with: { saveTokenAs: token1 }
+  tests: gta.expectResponseBodyToHaveProperty('email') # on get profile's response
+- GET: '{{baseUrl}}/orders'
+  headers: { Authorization: 'Bearer {{token1}}' }
+```
+
+Only one step of a set may have `useTests`, and only a set's steps.
+
 ### 2.6 Endpoint bases
 
 What every request to one endpoint gets, wherever the request is written. A file in
@@ -699,7 +767,8 @@ userId,expectedStatus,iterationLabel
 
 - **One run per row.** Every step runs for row 1, then every step again for row 2, and
   so on. Each run starts from a fresh variable scope, so nothing one row sets leaks into
-  the next. Three steps and two rows report six results.
+  the next, except a value set to last the run (§2.10). Three steps and two rows report
+  six results.
 - **Precedence.** A row's values sit over the environment and under what code sets
   (§4).
 - **`iterationLabel`**, an optional column, names the row in reports, as in
@@ -793,6 +862,41 @@ own, what the `command` prints, then overrides.
 
 Every report records the flag values a run used and where each came from.
 
+### 2.10 Setup, teardown and values that last the run
+
+`setup` and `teardown` are lists of steps, like `steps`, run once around it:
+
+```yaml
+id: approved-domains
+setup: # once, before the first row
+  - use: seed-admin # log in, grant an admin role, wait for it to land
+teardown: # once, after the last row, even when a row failed
+  - name: remove the grant
+    DELETE: '{{baseUrl}}/admins/{{adminId}}'
+steps: # once per row of approved-domains.csv, as before
+  - name: approve the domains
+    PUT: '{{baseUrl}}/orgs/{{orgId}}/domains'
+```
+
+- **Order.** Setup, then the steps once per data row (or once, with no data file), then
+  teardown. They work the same with or without a data file.
+- **What setup sets lasts the run.** Every value setup sets or captures is there for
+  every row and for teardown. It sits over the environment and under a row's values
+  (§4).
+- **A row keeps a value for the rows after it** with
+  `gta.set(name, value, { scope: 'run' })`. Anything else a row sets lasts only that
+  row, as §2.8 says.
+- **Setup failing stops the rows.** They are not run and count as skipped. Teardown
+  still runs.
+- **Teardown always runs**, every step of it, after failures and after `bail`. A run that
+  is cancelled, or stopped by `timeoutCollection`, stops where it is.
+- **The collection applies to them**: its headers, settings, `before`, `tests` and
+  flags, as to any step. Their steps may be use steps and may use `forEach`, but carry
+  no `tags`: they run whenever the collection does.
+- A request set, a base collection and an endpoints file have no setup or teardown.
+- Reports name their results `setup › log in` and `teardown › remove the grant`. Running
+  a single step in the desktop app does not run them.
+
 ---
 
 ## 3. Checking a response
@@ -817,6 +921,19 @@ tests: |
 A path addresses the body in dot or bracket notation: `user.name`; `groups.0.name` or
 `groups[0].name` for an index; and `sessions[].id` for a property of every item.
 
+A key that holds a `.`, `[` or `]`, or is empty, goes in brackets as a JSON string:
+`jwt.payload["https://example.com/id"]`, `modules[""].edition`. Reports show such keys
+the same way.
+
+As in xtest, a path can also be a list of keys: `['jwt', 'payload', 'https://example.com/id']`.
+Each item is one key, whatever it holds, and a number is its digits, so
+`['groups', 0, 'name']` reads an index. Every function that takes a path takes a list
+too, and so does `pathToProperty`.
+
+A path that runs into a `null` before its end, such as `phoneNumber.number` when
+`phoneNumber` is `null`, reads as that `null` for a check that the value is `null`, as
+xtest read it. For any other check the property is not present.
+
 ### Body conversion
 
 A JSON body is used as it is. An XML body is converted: the root element is the single
@@ -868,7 +985,15 @@ gta.expectResponseBodyToHaveUnorderedArray('users', [
 ```
 
 `gta.expectResponseBodyToHaveUnorderedArrayNotThisItem(path, list)` passes when no item
-matches.
+matches. A `compareValue` may be a `RegExp`.
+
+- **Each call prefers items an earlier call did not match.** Two calls with the same
+  description find two items when there are two, so each capture and strict validation
+  see a different one. A sort starts this over, since its indexes name other items.
+- **A list of one `notThisExpectedValue` entry reads as xtest read it.** Without strict
+  validation it means no item has that value, so an empty array passes. With it, it
+  means one item whose value is something else, as any list does. The step's last word
+  on strict validation decides.
 
 ### Strict validation
 
@@ -881,6 +1006,8 @@ checked, ignored or captured.
   vouch for what is inside.
 - It is judged after the collection's and the step's `tests` have both run, over what
   either checked.
+- A binary or HTML body has no properties to leave unchecked, so strict validation
+  passes. A check on one still fails.
 
 ### Sorting
 
@@ -922,10 +1049,15 @@ to. Whitespace inside the braces is ignored. In code, read a variable with
 
 - **Values keep their type.** A string that is exactly one reference returns the
   variable's own value, so a variable written as `true` arrives as a boolean. Anything
-  else is text, and `null` becomes the empty string inside it.
+  else is text, and `null` becomes the empty string inside it, except in a `json` body,
+  where it is written `null`, so `"website": {{site}}` stays valid JSON.
 - **An unknown variable is an error**, never literal text. The step fails in its own
   `interpolate` phase, naming the variable, and nothing is sent. A reference loop, or a
   chain more than 16 deep, fails the same way.
+- **`{{@name}}` reads the variable `name` names.** With `saveAs: token1`,
+  `{{@saveAs}}` is the value of `token1`. The name may itself be built from variables.
+  A name that is not text, or names no variable, fails like an unknown variable. A
+  request set uses it to read what it saved under its caller's name (§2.5).
 - **In YAML, quote a value that starts with `{{`.** Unquoted, YAML reads `{` as a map.
 
 ### Built-in variables
@@ -945,7 +1077,8 @@ Lowest precedence first:
 
 ```
 global project vars → project vars → base collection vars → collection vars
-  → environment → data file row → gta.set and captures → process environment
+  → environment → what lasts the run → data file row → gta.set and captures
+  → process environment
 ```
 
 The environment is the global project's file of that name, if there is one, with the
@@ -1006,13 +1139,19 @@ There is nothing to load, and no `startXTest` or `endXTest`.
 | `gta.test(name, fn)`: a named check that passes unless `fn` throws or rejects; `fn` may be async |                    |
 | `gta.get(name)`: a variable's current value                                                      |         ✓          |
 | `gta.set(name, value)`: set a variable for the rest of the run                                   |         ✓          |
+| `gta.set(name, value, { scope: 'run' })`: the same, lasting past this data row too (§2.10)       |         ✓          |
+| `gta.skip(reason?)`: send nothing for this step, and report it skipped with the reason           |         ✓          |
+| `gta.skipRest(reason?)`: skip the rest of this row's steps; in `before.script`, this one too     |         ✓          |
 | `gta.flag(name)`: a feature flag's value; an unknown flag is an error                            |         ✓          |
 | `gta.uuid()`: a random (version 4) UUID                                                          |         ✓          |
 | `gta.uuidv7()`: a time-ordered (version 7) UUID                                                  |         ✓          |
 | `gta.randomInt(min, max)`: a whole number, both ends included                                    |         ✓          |
 | `gta.date(format, secondsOffset = 0, timeZone = 'local')`                                        |         ✓          |
 
-Calling a response check in `before.script` is an error.
+Calling a response check in `before.script` is an error, and so is `gta.skip` in
+`tests`, where the request has been sent. After `gta.skip` the script runs to its end,
+and no later `before.script` runs. `gta.skipRest` skips steps of this row only: the next
+row, and teardown, still run.
 
 `gta.date` formats with strftime specifiers: `%Y %y %m %d %e %H %I %M %S %L %p %b %B %a
 %A %j %Z %z %s %F %T %%`. An unrecognized specifier is left in the output, so a typo is
@@ -1024,7 +1163,7 @@ xtest's military zone letters (`U` is -08:00, not UTC).
 | Global     | What it is                                                                                                                                                              |
 | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `res`      | `tests` only: `status`, `statusText`, `headers` (lower-cased names), `header(name)`, `body` (parsed JSON, converted XML, or text), `text`, `time` (ms), `size` (bytes). |
-| `req`      | `method`, `url`, `headers`, `body`: as sent in `tests`, as written in `before.script`.                                                                                  |
+| `req`      | `method`, `url`, `headers`, `body`: as sent in `tests`, as written in `before.script`, where a script may change `headers` and `body` (below).                          |
 | `assert`   | Node's strict `assert`, for use inside `gta.test`.                                                                                                                      |
 | `console`  | Captured into the step's result.                                                                                                                                        |
 | `params`   | A request set's params (§2.5), in its own scripts and in the tests of the use step running it.                                                                          |
@@ -1042,6 +1181,24 @@ run the same on any machine, and a request belongs in a step, where it is record
   marked as errored. Checks made before it are kept.
 - A script that runs for more than 10 seconds is stopped.
 
+**Changing the request in `before.script`.** `req` there is the request as written,
+`{{variables}}` still in it. A script may change it before it is sent, and a later
+`before.script` sees the change:
+
+```js
+const claims = JSON.parse(req.body)
+if (!params.crmContactId) delete claims.crm_contact_id // leave the member out
+req.body = JSON.stringify(claims)
+req.headers['X-Trace'] = '{{traceId}}'
+delete req.headers['X-Debug']
+```
+
+- `req.body` is text, and can be changed for a `json`, `xml`, `text` or `graphql` body.
+  A form, multipart or file body is built from its parts, so it cannot.
+- `req.headers` is a map of names to values: add, change or delete them.
+- Variables in what the script writes resolve afterwards, as in the file.
+- Anything else is a pre-request error, and nothing is sent.
+
 ### Check files
 
 `checks/*.js` in a project, and in its global project, hold functions every script can
@@ -1154,7 +1311,8 @@ it. Rules checked at run time fail the step, or the run, before anything is sent
   ignoring case.
 - There is no `name` key (use `id`).
 - A step has `tags` only if the collection has `stepTags: true`.
-- With `params`: no step is a use step, and there is no `vars`.
+- With `params`: no step is a use step, and there is no `vars`, `setup` or `teardown`.
+- A `setup` or `teardown` step has no `tags`.
 - A data file, when present, reads (§2.8).
 
 **Steps**
@@ -1167,6 +1325,8 @@ it. Rules checked at run time fail the step, or the run, before anything is sent
 - `before` holds only `script`. `before.set` is not supported.
 - There is no `expect:` key; checks go in `tests`.
 - `base` is only ever `false`.
+- `forEach` is a string, and a use step has none.
+- `useTests` is only ever `true`, only on a request set's step, and on one step at most.
 
 **Values**
 
@@ -1189,9 +1349,10 @@ it. Rules checked at run time fail the step, or the run, before anything is sent
 **Library files**
 
 - A request set (`requests/`) has `params`, uses no other set, and has no `vars`.
-- A base collection (`bases/`) has no `steps` or `params`, and no `extends`.
+- A base collection (`bases/`) has no `steps`, `setup`, `teardown` or `params`, and no
+  `extends`.
 - An endpoint (`endpoints/`) has a URL that is a path starting with `/`, and no use
-  steps.
+  steps. An endpoints file has no `setup` or `teardown`.
 - A check file's name is a JavaScript identifier; if it isn't, the file is not loaded.
 - `collections/`, `requests/`, `endpoints/` and `bases/` hold files at most one directory
   down.
@@ -1222,7 +1383,13 @@ it. Rules checked at run time fail the step, or the run, before anything is sent
 - The flag `command`, if any, has every quote closed, starts, and exits `0` within 60
   seconds, printing a JSON object.
 - Every file a body names can be read from the project folder.
-- Every `use:` and `extends:` names a usable file.
+- Every `use:` and `extends:` names a usable file, and every `with:` suits its set.
+  `gta get` checks these without running anything (§1.3).
+- A step's `forEach` resolves to a JSON array.
+- In `{{@name}}`, `name` holds text naming a variable that exists.
+- A `before.script` sets `req.body` to text, and only for a `json`, `xml`, `text` or
+  `graphql` body; `req.headers` stays a map.
+- `gta.set`'s `scope`, when given, is `'run'`.
 
 ---