saffron-ai 1.3.4 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +745 -2
- package/README.md +18 -4
- package/dist-pkg/cli.js +450 -425
- package/package.json +26 -2
- package/skills/saffron/SKILL.md +40 -7
- package/skills/saffron/references/api.md +8 -1
- package/skills/saffron/references/config.md +5 -4
- package/skills/saffron/references/database.md +164 -0
- package/skills/saffron/references/syntax.md +32 -2
- package/templates/playwright/saffron-helpers.ts +346 -69
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "saffron-ai",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.4.0",
|
|
4
4
|
"description": "End-to-end tests written in plain language: an AI agent records each scenario once, replays run at zero tokens, and broken replays heal into proposals you review",
|
|
5
5
|
"scripts": {
|
|
6
6
|
"test": "vitest run",
|
|
@@ -47,9 +47,14 @@
|
|
|
47
47
|
"devDependencies": {
|
|
48
48
|
"@cursor/sdk": "^1.0.32",
|
|
49
49
|
"@openai/codex-sdk": "^0.158.0",
|
|
50
|
+
"@types/mssql": "^12.3.0",
|
|
50
51
|
"@types/node": "^26.1.0",
|
|
52
|
+
"@types/pg": "^8.23.1",
|
|
51
53
|
"esbuild": "^0.28.1",
|
|
52
54
|
"marked": "^18.0.5",
|
|
55
|
+
"mssql": "^12.7.14",
|
|
56
|
+
"mysql2": "^3.24.6",
|
|
57
|
+
"node-sql-parser": "^5.4.0",
|
|
53
58
|
"typescript": "^6.0.3",
|
|
54
59
|
"vitest": "^4.1.9"
|
|
55
60
|
},
|
|
@@ -87,5 +92,24 @@
|
|
|
87
92
|
"README.md",
|
|
88
93
|
"CHANGELOG.md",
|
|
89
94
|
"LICENSE"
|
|
90
|
-
]
|
|
95
|
+
],
|
|
96
|
+
"optionalDependencies": {
|
|
97
|
+
"pg": "^8.23.1"
|
|
98
|
+
},
|
|
99
|
+
"peerDependencies": {
|
|
100
|
+
"node-sql-parser": "^5.4.0",
|
|
101
|
+
"mssql": "^12.7.14",
|
|
102
|
+
"mysql2": "^3.24.6"
|
|
103
|
+
},
|
|
104
|
+
"peerDependenciesMeta": {
|
|
105
|
+
"node-sql-parser": {
|
|
106
|
+
"optional": true
|
|
107
|
+
},
|
|
108
|
+
"mssql": {
|
|
109
|
+
"optional": true
|
|
110
|
+
},
|
|
111
|
+
"mysql2": {
|
|
112
|
+
"optional": true
|
|
113
|
+
}
|
|
114
|
+
}
|
|
91
115
|
}
|
package/skills/saffron/SKILL.md
CHANGED
|
@@ -3,14 +3,14 @@ name: saffron
|
|
|
3
3
|
description: Create, translate, or edit Saffron test scenarios in .saffron or .feature files, browser UI tests and HTTP API tests (@api) alike, including validating their recordings and replay. Use when asked to author or fix these test files in a Saffron project. Do not activate for ordinary application development, generic Cucumber projects, or requests only to run existing tests, inspect reports, review proposals, or configure Saffron.
|
|
4
4
|
license: Saffron Free Use License v1.0 (see the saffron-ai package LICENSE)
|
|
5
5
|
metadata:
|
|
6
|
-
saffron-ai: "1.
|
|
6
|
+
saffron-ai: "1.4.0"
|
|
7
7
|
author: saffron-ai
|
|
8
8
|
homepage: https://saffron-ai.io
|
|
9
9
|
---
|
|
10
10
|
|
|
11
11
|
# Saffron
|
|
12
12
|
|
|
13
|
-
This skill describes saffron-ai 1.
|
|
13
|
+
This skill describes saffron-ai 1.4.0 (see the compatibility page: https://saffron-ai.io/docs/compatibility).
|
|
14
14
|
|
|
15
15
|
Apply this workflow only when asked to create, translate, or edit Saffron
|
|
16
16
|
scenarios in `.saffron` or `.feature` files. A Saffron mention or an
|
|
@@ -96,14 +96,17 @@ end up with something recorded against it. Put
|
|
|
96
96
|
the final assertions **last**; the trailing block of `Then` steps is
|
|
97
97
|
strict under every policy. Use `Given`/`When` for actions. A text check
|
|
98
98
|
passes only on text the page shows: hidden elements, hidden parts of an
|
|
99
|
-
element
|
|
100
|
-
means visible.
|
|
99
|
+
element, hidden dropdown options and frames the page hides never count,
|
|
100
|
+
so "should be shown" means visible. One narrow exception to "never healed": a database check
|
|
101
|
+
whose query fails after a schema change may have its query (table,
|
|
102
|
+
columns, joins) repaired by the agent; its binds, expected rows and row
|
|
103
|
+
count never change.
|
|
101
104
|
|
|
102
105
|
## 3. Never bake volatile or secret values into files
|
|
103
106
|
|
|
104
107
|
- Secrets: `{env:VAR}`, as in `When I enter "{env:ADMIN_PASSWORD}" in the password field`.
|
|
105
108
|
- Shared test data (accounts, names, enum values): put it in `data/*.json` and write `{data:users.admin.email}` (file, then dotted path; one value, not a list). The recording stores the token, so editing the file changes the next replay at zero tokens. Never for secrets: data files are committed. To check a whole list (an enum), point a Then step at it: `Then the status filter should list every {data:enums.OrderStatus}`; replay iterates the file's list, so adding a value needs no re-record. `Examples: {data:roles}` reads outline rows from `data/roles.csv` or a JSON list (`.saffron` only). A value that must be new on every run (an email for a sign-up) is `{unique:name}`: fresh per run, the same within it.
|
|
106
|
-
- State the application must be in before the suite (seeded users, a reset database): do not script it through the UI in every scenario. The project's `saffron.config.json` takes `"setup"` and `"teardown"` shell commands, run once around `saffron run` (teardown always runs). Propose a seed script there; keep scenario steps for the behaviour under test.
|
|
109
|
+
- State the application must be in before the suite (seeded users, a reset database): do not script it through the UI in every scenario. The project's `saffron.config.json` takes `"setup"` and `"teardown"` shell commands, run once around `saffron run` (teardown always runs). Propose a seed script there; keep scenario steps for the behaviour under test. Data one scenario needs can be seeded inside it instead with a database seed step (`Given a customer "ann-{unique:c}@example.test" on the "basic" tier`), when the project configures `db` with writes on; see Database steps below.
|
|
107
110
|
- Dates: write intent when the day should move ("1 day from today" records as `{date+1}`); a date you write out (`I enter 2026-10-10`, in a table, a data file or `{env:...}`) is entered as written on every run. An Examples cell such as `{date+2:DD.MM.YYYY}` gives each row its own day. A date the app shows in words or as a range is a token too (`{date+7:D. MMM:de-DE}`, `{date+7 - +9:MMM D, YYYY}`), never a setup script.
|
|
108
111
|
- Dynamic display values: capture and compare, e.g. `I record the total as "first"` … `"first" should differ from the displayed total`.
|
|
109
112
|
|
|
@@ -296,6 +299,28 @@ or "should match the contract", and name the status of an error case.
|
|
|
296
299
|
Pin the contract first (`npx saffron api pin <url or file>`), and read
|
|
297
300
|
[API tests](references/api.md) for the recording, review and run details.
|
|
298
301
|
|
|
302
|
+
## Database steps (beta)
|
|
303
|
+
|
|
304
|
+
```gherkin
|
|
305
|
+
Scenario: An upgrade is saved
|
|
306
|
+
Given a customer "ann-{unique:c}@example.test" on the "basic" tier
|
|
307
|
+
When I upgrade "ann-{unique:c}@example.test" to "gold" on the account page
|
|
308
|
+
Then the customer "ann-{unique:c}@example.test" should be on the "gold" tier in the database
|
|
309
|
+
And no refund should exist in the database for "ann-{unique:c}@example.test"
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
With `db` in `saffron.config.json` (PostgreSQL, MySQL or SQL Server), a `Then` can check
|
|
313
|
+
what the database holds and a `Given` or `When` can seed data, in browser and `@api`
|
|
314
|
+
scenarios alike. Name the database in a check ("in the database") and name
|
|
315
|
+
the record by a value the scenario wrote: the step's text, an Examples
|
|
316
|
+
column, a `{unique:...}` value, or a value an earlier step kept. A check
|
|
317
|
+
bound to nothing ("a customer exists") is refused while recording. A
|
|
318
|
+
"not in the database" check waits out `db.timeout` (3 s by default) before
|
|
319
|
+
it passes. Seeding needs `"writes": true` and the database's host or name
|
|
320
|
+
in `allow`; seed what the app gives no way to set, with `{unique:...}`
|
|
321
|
+
values, and the rows are undone when the scenario ends. Read
|
|
322
|
+
[Database steps](references/database.md) for the details.
|
|
323
|
+
|
|
299
324
|
## Repeated step groups → StepSets (`.saffron` files only)
|
|
300
325
|
|
|
301
326
|
`.saffron` is a superset of `.feature`: rename the file, nothing else
|
|
@@ -353,7 +378,10 @@ refused (red, at zero tokens) with guidance naming the column and the
|
|
|
353
378
|
step. That includes dates without a year and one- or two-character
|
|
354
379
|
values. A blank cell is refused the same way, or, when its step types
|
|
355
380
|
`<column>`, the first filled row records once more and serves the blank
|
|
356
|
-
rows too.
|
|
381
|
+
rows too. A value the page sets in its own way (a phone or IBAN in
|
|
382
|
+
blocks, an id zero-padded inside a longer one like `INV-000104`) binds the
|
|
383
|
+
recording to its row the same way: give the cell the value as the page
|
|
384
|
+
shows it. Keep a check that lists all rows' values out of the Outline.
|
|
357
385
|
Tests live in the folder named by `features` in `saffron.config.json`
|
|
358
386
|
(`features/` by default).
|
|
359
387
|
|
|
@@ -392,6 +420,7 @@ Use Saffron itself first: `{date...}` tokens (ranges and locales included), capt
|
|
|
392
420
|
| Create a fixed record (`"cust-1"`) in an API step | `{unique:name}`: every run, and the proof replay, creates its own |
|
|
393
421
|
| Check an API response's status in a `When` | Name the status in a `Then` ("should answer 404") |
|
|
394
422
|
| One long step line carrying an API key, ids and several quoted parameters | A short step and a 2-column data table under it, one value per row |
|
|
423
|
+
| A database check bound to nothing ("Then a customer exists in the database") | Name the record the scenario made: `Then the customer "ann-{unique:c}@example.test" should be saved in the database` |
|
|
395
424
|
|
|
396
425
|
## References
|
|
397
426
|
|
|
@@ -399,6 +428,10 @@ Use Saffron itself first: `{date...}` tokens (ranges and locales included), capt
|
|
|
399
428
|
outlines, secrets, dates, network waits, page furniture, StepSets.
|
|
400
429
|
- [Configuration & CI](references/config.md): `saffron.config.json`,
|
|
401
430
|
flags, `--strict`, cross-browser and browser matrix, devices, workers
|
|
402
|
-
and agent workers
|
|
431
|
+
and agent workers (`@serial` keeps scenarios that share data apart),
|
|
432
|
+
heal model, secrets setup, the API settings.
|
|
403
433
|
- [API tests](references/api.md): `@api` scenarios, the pinned contract,
|
|
404
434
|
writing and reviewing request steps, API calls from a browser scenario.
|
|
435
|
+
- [Database steps](references/database.md): checks and seeds against the
|
|
436
|
+
configured database, what is recorded and refused, cleanup, and the one
|
|
437
|
+
Then whose query may be repaired.
|
|
@@ -2,7 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
A scenario tagged `@api` tests an HTTP API with no browser. The agent
|
|
4
4
|
records each request once, every later run replays it at zero tokens, and
|
|
5
|
-
`Then` steps are never healed, as for browser scenarios
|
|
5
|
+
`Then` steps are never healed, as for browser scenarios (the one
|
|
6
|
+
exception is a database check's query after a schema change, see
|
|
7
|
+
[Database steps](database.md)). A browser
|
|
6
8
|
scenario can also call the API in a step; see the end of this page.
|
|
7
9
|
|
|
8
10
|
## Before writing
|
|
@@ -90,6 +92,11 @@ Feature: Customers API
|
|
|
90
92
|
2xx is recorded only when its step names that status; otherwise the
|
|
91
93
|
step fails, so a failure seen while recording never replays as the
|
|
92
94
|
expected result.
|
|
95
|
+
- **A `Then` says what it checks.** A `Then` whose only check would be
|
|
96
|
+
the 2xx class, or the status its request already required, is refused
|
|
97
|
+
(it could never fail): name a field or "should match the contract". A
|
|
98
|
+
time limit ("within 2 seconds") or a poll's limit is recorded exactly as
|
|
99
|
+
the step states it in figures.
|
|
93
100
|
- **Secrets stay `{env:VAR}` tokens**, typically in a step that names the
|
|
94
101
|
credential: `Given I use the API token "{env:API_TOKEN}"`. The request
|
|
95
102
|
sends the value and the recording keeps the token.
|
|
@@ -44,7 +44,7 @@ your-project/
|
|
|
44
44
|
|
|
45
45
|
| Key | Meaning |
|
|
46
46
|
|---|---|
|
|
47
|
-
| `dataDir` | Folder of JSON files read by `{data:file.key}` tokens (default `data`). Committed; not for secrets |
|
|
47
|
+
| `dataDir` | Folder of JSON and CSV files read by `{data:file.key}` tokens and by `Examples: {data:file}` (default `data`). Committed; not for secrets |
|
|
48
48
|
| `baseURL` | App under test; steps say "the login page", not full URLs |
|
|
49
49
|
| `storageState` | Playwright storage-state JSON so replays and the agent start authenticated (`npx playwright open --save-storage=.auth/state.json <url>`) |
|
|
50
50
|
| `sessions` | Per browser session (`As "buyer" ...`): `{ "buyer": { "storageState": ".auth/buyer.json" } }` starts that session signed in as its own person; others, and the default session, take `storageState` |
|
|
@@ -64,7 +64,7 @@ your-project/
|
|
|
64
64
|
| `device` / `viewport` | A Playwright device profile (`"iPhone 15"`) and/or a `"WIDTHxHEIGHT"` size for every scenario; a desktop recording replays under it or fails at the step that differs |
|
|
65
65
|
| `timezone` | IANA zone (`"Europe/Berlin"`) the browser runs in and `{date...}` tokens count days in (default: the machine's) |
|
|
66
66
|
| `dateLocale` | Locale of month and weekday names in `{date...}` tokens that name none (default `en-US`) |
|
|
67
|
-
| `workers` | Parallel replay workers |
|
|
67
|
+
| `workers` | Parallel replay workers; scenarios tagged `@serial` or `@serial:<name>` still run one at a time per name |
|
|
68
68
|
| `agentWorkers` | Agent sessions (record, heal) that may run at once, each in its own browser (default 1; each is a paid session) |
|
|
69
69
|
| `shardBy` | How `--shard` splits the suite: `"scenario"` (default) or `"file"` (each feature file on one machine) |
|
|
70
70
|
| `provider` | Whose agent records and heals, on its login on this machine: `claude` (default), `codex` (ChatGPT plan), `antigravity` (Google AI Pro/Ultra via the `agy` CLI, experimental), `cursor`. Also `--provider` |
|
|
@@ -72,7 +72,8 @@ your-project/
|
|
|
72
72
|
| `healModel` | Cheaper model for heal sessions only |
|
|
73
73
|
| `antigravityApiKey` | `true`: Antigravity signs in with `GEMINI_API_KEY` even beside an agy login (default: a login wins) |
|
|
74
74
|
| `ci` | Where `saffron runs` and `import --run` find CI runs: `{"provider": "github" \| "azure", "organization": ..., "project": ...}`. Unset, an Azure Repos remote means Azure Pipelines (through `az`), anything else GitHub Actions (through `gh`). `organization` and `project` are for a GitHub repository built by Azure Pipelines. `cacheRuns` and `cacheMB` cap the runs viewed into `.saffron/ci/` (default 10 and 1024) |
|
|
75
|
-
| `
|
|
75
|
+
| `db` | The database that database steps check and seed (PostgreSQL, MySQL or SQL Server, by the URL's scheme; `npm install node-sql-parser` and `pg`, `mysql2` or `mssql`): `{"url": "{env:DATABASE_URL}", "writes": false, "allow": ["localhost"]}`. `url` must be an `{env:...}` reference; `writes` (default `false`) lets `Given` and `When` steps insert and update with `dbWrite`; `allow` lists the hosts or database names a write may go to; `timeout` (default 3000 ms) is how long a check reads again, and a "not in the database" check must hold that long; `cleanup` is `"all"` (default: delete inserts, put updated rows back), `"inserts"` or `"none"`. A `Then` records `expectRows`: one SELECT, values as `$n` binds bound to a scenario value, checks on its rows; a check whose query fails on the schema may have its query repaired by the agent, its binds, checks and row count unchanged |
|
|
76
|
+
| `api` | The API under test: `{"baseURL": ..., "contract": "api/openapi.json", "transcript": "on-failure"}`. `baseURL` is where API steps' relative paths go (unset: the run's `baseURL`); `contract` is the pinned OpenAPI 3 document schema checks validate against (`saffron api pin` writes it), and a change to it, or to which file it names, makes the recordings checked against it stale; `transcript` keeps each scenario's requests and responses for the report and `saffron trace`: `"on-failure"` (default), `"on"` or `"off"` (statuses and timings only); `services` names more APIs, each `{"baseURL": ..., "contract": ...}`, which a request names instead of a host (`--api-url name=<url>` points one elsewhere for a run). A key Saffron does not read under `api` or a service is refused. See [API tests](api.md) |
|
|
76
77
|
|
|
77
78
|
## CLI
|
|
78
79
|
|
|
@@ -90,7 +91,7 @@ your-project/
|
|
|
90
91
|
| `saffron mcp` / `saffron lsp` | MCP tools (`search_steps`, `list_step_sets`, `project_status`, `run_plan`, `api_contract`, `verify`) for AI assistants / language server for editors |
|
|
91
92
|
| `saffron api pin <source> [--out file] [--service name]` | Pin the API contract: fetch an OpenAPI 3 document from a Swagger UI page, a document URL or a file, and write it as JSON (default `api/openapi.json`, or `api/<name>.json` with `--service`). Add it to the config as `api.contract`, or as `api.services.<name>.contract` |
|
|
92
93
|
| `saffron init [--examples] [--agents list]` | Install this skill into the project's agent directories, register MCP, add an AGENTS.md block, scaffold config; `--examples` adds the Saucedemo example suite |
|
|
93
|
-
| `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs and 1 GB unless `ci.cacheRuns` and `ci.cacheMB` say otherwise), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
|
|
94
|
+
| `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line; it exits 1 when the merged run has a red (or a pending review with `strict` on). `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs and 1 GB unless `ci.cacheRuns` and `ci.cacheMB` say otherwise), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
|
|
94
95
|
| `saffron dashboard [--stdout] [--no-open]` | The suite's quality from the run history (failing and for how long, flaky, repeatedly healed, never run, slowest, trends, per tag); writes and opens `.saffron/reports/dashboard.html`, `--stdout` prints it |
|
|
95
96
|
| `saffron export [-o file] [--branch name [--base branch]] [--report-only]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). `--report-only` writes a summary instead (report and screenshots only, `saffron-summary.zip`): upload it as a `saffron-summary*` artifact on every run so any run's results can be viewed; a run that left only summaries has nothing to import. In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
|
|
96
97
|
| `saffron runs [--branch b] [--limit n] [--provider github\|azure] [--json]` | The branch's CI runs, with how many of each run's proposals are pending here; exit 2 when `gh` or `az` is missing or signed out |
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# Database steps (beta)
|
|
2
|
+
|
|
3
|
+
A step can check what the database holds, or seed the data a scenario
|
|
4
|
+
starts from, with no query written by hand. While recording, the agent
|
|
5
|
+
reads the schema, writes one parameterized statement, runs it, and records
|
|
6
|
+
it with the step's values as binds; every later run replays it at zero
|
|
7
|
+
tokens. They work in browser scenarios and in `@api` scenarios alike.
|
|
8
|
+
PostgreSQL, MySQL and SQL Server, named by the URL's scheme
|
|
9
|
+
(`postgres://`, `mysql://`, `mssql://`).
|
|
10
|
+
|
|
11
|
+
## Setup
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{
|
|
15
|
+
"db": { "url": "{env:DATABASE_URL}", "writes": false, "allow": ["localhost", "shop_test"], "timeout": 3000, "cleanup": "all" }
|
|
16
|
+
}
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
- `npm install node-sql-parser` and the engine's driver (`pg`, `mysql2` or
|
|
20
|
+
`mssql`) in the project. Both load only when `db` is set.
|
|
21
|
+
- Every engine takes `$1`, `$2` binds; write the engine's own SQL
|
|
22
|
+
otherwise. On MySQL and SQL Server an `INSERT` with an auto-numbered key
|
|
23
|
+
writes one row (several rows give the key as binds), and an `UPDATE`
|
|
24
|
+
never sets the key: Saffron reads the rows back by it. A recording is
|
|
25
|
+
refused on another engine.
|
|
26
|
+
- `url` must be an `{env:...}` reference: the connection string and its
|
|
27
|
+
password never sit in a file, and never reach a recording, a proposal,
|
|
28
|
+
a report, a trace or the agent. Write the host and the database in the
|
|
29
|
+
URL itself; a `?host=` parameter, or no host, is refused for writes.
|
|
30
|
+
- `writes` (default `false`) lets `Given` and `When` steps seed data.
|
|
31
|
+
- `allow` lists the hosts or database names a write may go to. A write
|
|
32
|
+
anywhere else is refused before it runs.
|
|
33
|
+
- `timeout` (milliseconds, default 3000) is how long a check reads again.
|
|
34
|
+
- `cleanup` (default `"all"`) is what is undone when a scenario ends.
|
|
35
|
+
- Give Saffron its own database user on a test database.
|
|
36
|
+
|
|
37
|
+
## Wording the steps
|
|
38
|
+
|
|
39
|
+
```gherkin
|
|
40
|
+
Scenario: A new customer is saved
|
|
41
|
+
Given a customer "ann-{unique:c}@example.test" on the "basic" tier
|
|
42
|
+
When I upgrade "ann-{unique:c}@example.test" to "gold" on the account page
|
|
43
|
+
Then the customer "ann-{unique:c}@example.test" should be on the "gold" tier in the database
|
|
44
|
+
And no refund should exist in the database for "ann-{unique:c}@example.test"
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
- Name the database in a check (`in the database`), and name the record
|
|
48
|
+
by a value the scenario wrote: the step's own text, an Examples column,
|
|
49
|
+
a `{unique:...}` value, or a value an earlier step kept. A check that
|
|
50
|
+
names nothing ("a customer exists") would pass on rows any earlier run
|
|
51
|
+
left behind, and is refused while recording. A value spelled in the
|
|
52
|
+
step is the same every run: where the check must prove this run saved
|
|
53
|
+
the row, use a `{unique:...}` value or an id an earlier step kept.
|
|
54
|
+
- Seed with `{unique:...}` values where a column is unique: parallel
|
|
55
|
+
workers and proof replays each write their own.
|
|
56
|
+
- A seed `Given` states the data, not how to insert it. Seed only what the
|
|
57
|
+
app gives no way to set; a `When` that the app performs goes through the
|
|
58
|
+
app.
|
|
59
|
+
- A `Fixture:` block does not write to the database: seed in each
|
|
60
|
+
scenario's `Given`.
|
|
61
|
+
|
|
62
|
+
## What is recorded
|
|
63
|
+
|
|
64
|
+
| Action | Step | Holds |
|
|
65
|
+
|---|---|---|
|
|
66
|
+
| `dbWrite` | `Given`, `When` | One `INSERT`, or one `UPDATE` with a `WHERE`; values as `$1`, `$2` binds (`body`); `extract` keeps a field of the first row written as `{captured:name}` |
|
|
67
|
+
| `expectRows` | `Then` | One `SELECT`; values as binds; `json` checks on its rows read as a JSON list (`count`, `/0/column` equals, `item`); `rowCount`, the rows it found when recorded |
|
|
68
|
+
|
|
69
|
+
- Values are binds, never written into the SQL: a literal is refused, and
|
|
70
|
+
so is a comment. Each bind is compared with a column (`email = $1`); a
|
|
71
|
+
query that returns a bind, compares values with no column, or adds
|
|
72
|
+
`OR true` is refused. The `WHERE` must tie its rows to a bind with `=`,
|
|
73
|
+
`IN`, `LIKE` (or `ILIKE` on PostgreSQL). Each side of an `OR`, and
|
|
74
|
+
every value of an `IN`, needs such a bind (`email = $1 OR role =
|
|
75
|
+
'admin'` is refused); one side of an `AND` is enough. `<>`, `NOT (...)`, a
|
|
76
|
+
range alone, a bind in `LIMIT`, and `UNION` are refused, and a subquery
|
|
77
|
+
must tie its own rows. A `LIKE` value made only of `%` or `_` (an
|
|
78
|
+
Examples cell too) matches every row, so it is refused unless another
|
|
79
|
+
condition ties the rows. A JSON field can tie a check
|
|
80
|
+
(`data->>'orderRef' = $1`). Cast with `CAST($1 AS type)`, not `::`. A
|
|
81
|
+
`WITH` that selects a bind as a column is refused like a `SELECT $1`.
|
|
82
|
+
On MySQL join texts with `CONCAT()`: `||` runs as `OR` there and is
|
|
83
|
+
refused.
|
|
84
|
+
Binds are written as their tokens, so each Examples row checks its own
|
|
85
|
+
value. Bind a row's whole value (`<email>`), never a part of it ("ann"
|
|
86
|
+
of ann@example.test): a part is kept as written for every row and is
|
|
87
|
+
refused, unless every row holds it alike. An aggregate with no `GROUP BY` is always one row: check its
|
|
88
|
+
value (`/0/n equals 1`, or a count on a list it holds), never a count
|
|
89
|
+
or an exists on the row itself, also when the aggregate is in the
|
|
90
|
+
query's only `FROM` or `WITH` source. A count above 0, or a check that
|
|
91
|
+
there is no second row, under `LIMIT`, `TOP`, `FETCH`, `DISTINCT` or a
|
|
92
|
+
`GROUP BY` with no aggregate is refused, also when they sit in a query
|
|
93
|
+
read `FROM` or `WITH`, and so is any check under `OFFSET`, and `LIMIT
|
|
94
|
+
0`, `TOP 0` and `FETCH ... 0`. A field that must not be in a row
|
|
95
|
+
(`/0/data/discount`) holds only when that row is there.
|
|
96
|
+
- A check reads in a transaction that is rolled back (read-only on
|
|
97
|
+
PostgreSQL and MySQL). A date, a timestamp, a bigint or a numeric reads
|
|
98
|
+
as text in UTC (`2026-11-07`, `2026-11-07 09:00:00+00` on PostgreSQL),
|
|
99
|
+
the same on every machine; SQL Server reads a DECIMAL as a number.
|
|
100
|
+
- A presence check passes as soon as its rows are there, within
|
|
101
|
+
`db.timeout`. A "not in the database" check (a count of 0, a `count(*)` checked as 0, a field that
|
|
102
|
+
must not exist) waits for a row a worker deletes late, then must hold to
|
|
103
|
+
the end of the window, so a row a worker writes late fails it. A check holding both waits for the first part, then holds
|
|
104
|
+
the second to the end.
|
|
105
|
+
- The run report shows each check's SQL, row count and first five rows,
|
|
106
|
+
masked.
|
|
107
|
+
|
|
108
|
+
## Refused
|
|
109
|
+
|
|
110
|
+
`DELETE`, `TRUNCATE`, `DROP`, `ALTER`, `GRANT`, several statements in one,
|
|
111
|
+
a write inside a `WITH`, an `UPDATE` without a `WHERE`, `UPDATE ... FROM`,
|
|
112
|
+
`INSERT ... ON CONFLICT` and `ON DUPLICATE KEY UPDATE`, MySQL's
|
|
113
|
+
`INSERT IGNORE`, an `UPDATE` with a `JOIN`, a second table, `ORDER BY` or
|
|
114
|
+
`LIMIT`, a write to a table of another database (`otherdb.t`,
|
|
115
|
+
`db.schema.t`), table hints (`WITH (NOLOCK)`, `READPAST`),
|
|
116
|
+
`SELECT ... INTO`, MySQL's `/*! ... */` comments, a backslash before a
|
|
117
|
+
quote on SQL Server, a write in a `Then`, a write through a view or synonym, a MySQL read
|
|
118
|
+
that locks rows (`FOR UPDATE`, `FOR SHARE`, `LOCK IN SHARE MODE`), an `UPDATE` whose `WHERE`
|
|
119
|
+
binds nothing the scenario wrote, and any statement calling a function
|
|
120
|
+
whose effects reach past its transaction (`pg_terminate_backend`,
|
|
121
|
+
`dblink`, `set_config`, sequences, advisory locks, large objects, files, `SLEEP`, `BENCHMARK`, `LOAD_FILE`, `OPENQUERY`,
|
|
122
|
+
`OPENROWSET`).
|
|
123
|
+
Statements are judged on their parsed form, never their text.
|
|
124
|
+
|
|
125
|
+
## Cleanup
|
|
126
|
+
|
|
127
|
+
When a scenario ends, green or red (and at Ctrl-C or SIGTERM), Saffron undoes its
|
|
128
|
+
writes, the latest first, by primary key and nothing else:
|
|
129
|
+
|
|
130
|
+
| `cleanup` | Inserts | Updates |
|
|
131
|
+
|---|---|---|
|
|
132
|
+
| `"all"` (default) | deleted | changed columns put back as they were before the step |
|
|
133
|
+
| `"inserts"` | deleted | left |
|
|
134
|
+
| `"none"` | left | left, for debugging |
|
|
135
|
+
|
|
136
|
+
A write that must be undone needs a table with a primary key; an `UPDATE`
|
|
137
|
+
that changes a primary key is refused under `"all"`. What a recording
|
|
138
|
+
wrote is undone before its proof replays it, and what the proof wrote
|
|
139
|
+
before the stability and row proofs. A row cleanup cannot undo is named
|
|
140
|
+
on the console. A killed run (SIGKILL, out of memory) undoes nothing, and
|
|
141
|
+
rows a database trigger writes are never undone: sweep them in
|
|
142
|
+
`teardown`. `run --plan` lists a write that `cleanup` keeps as a side
|
|
143
|
+
effect.
|
|
144
|
+
|
|
145
|
+
## The one Then that may be repaired
|
|
146
|
+
|
|
147
|
+
A `Then` is never healed, with one exception. When a database check's
|
|
148
|
+
stored query fails because the schema changed (a table or column gone, a
|
|
149
|
+
type that no longer fits), the agent repairs where the query looks: its
|
|
150
|
+
table, columns and joins. Its binds, its expected rows and its row count
|
|
151
|
+
are locked: a repair that changes any of them, or adds a `LIMIT`,
|
|
152
|
+
`DISTINCT`, `GROUP BY` or aggregate, is refused, and the step stays red.
|
|
153
|
+
A check that expects no rows, a `count(*)` of 0, a count from an Outline
|
|
154
|
+
column, or a field that must not be there, is not repaired: record it again.
|
|
155
|
+
A repaired check, and a heal that adds or changes a database write, is
|
|
156
|
+
left out of `accept --all`: it is accepted by file after review. The repaired check must hold on the live data and on a proof
|
|
157
|
+
replay; the step is reported healed and a proposal names the old and new
|
|
158
|
+
SQL. A query that runs and finds other rows is plain red, and no agent is
|
|
159
|
+
started.
|
|
160
|
+
|
|
161
|
+
## Not yet
|
|
162
|
+
|
|
163
|
+
MongoDB; `saffron eject` writes these scenarios as `test.fixme`; the
|
|
164
|
+
editor plugins show database steps only as the runner reports them.
|
|
@@ -48,6 +48,13 @@ Scenario Outline: Failed login shows an error
|
|
|
48
48
|
| standard | wrong | Wrong username or password |
|
|
49
49
|
```
|
|
50
50
|
|
|
51
|
+
A `<name>` within an edit or two of a column (a misspelt `<rol>` beside
|
|
52
|
+
`| role |`) is a parse error naming the columns; other tags, such as
|
|
53
|
+
`"<script>"` or `List<String>`, stay literal. An Outline with no Examples
|
|
54
|
+
whose steps use a placeholder is refused (add Examples or use
|
|
55
|
+
`Scenario:`), and so is an Examples block with no table
|
|
56
|
+
(`Examples: data:roles`; the file form is `Examples: {data:roles}`).
|
|
57
|
+
|
|
51
58
|
## Data tables
|
|
52
59
|
|
|
53
60
|
**2-column = key/value.** Values record as `<table:key>` references:
|
|
@@ -171,7 +178,8 @@ is 9 digits.
|
|
|
171
178
|
`{env:...}`) is entered as written on every run. In an Outline, each
|
|
172
179
|
Examples row's `{date+N}` cell, in any format, gives that row its own day.
|
|
173
180
|
- Text assertions read only what the page shows: hidden text, hidden
|
|
174
|
-
options
|
|
181
|
+
options, hidden parts of an element and anything inside a frame the
|
|
182
|
+
page hides never satisfy them.
|
|
175
183
|
- Capture and compare displayed values instead of literals:
|
|
176
184
|
|
|
177
185
|
```gherkin
|
|
@@ -251,6 +259,10 @@ When I open a new tab at the admin page
|
|
|
251
259
|
Recorded as `switchTab` / `closeTab` / `newTab` with 0-based tab indexes
|
|
252
260
|
in creation order; replay repeats them in order. Say which tab in the
|
|
253
261
|
step ("the first tab", "the preview tab") so the recording is unambiguous.
|
|
262
|
+
When the steps after the click keep working on the page clicked, the
|
|
263
|
+
recording switches back to it, so replay runs them there, not on the new
|
|
264
|
+
tab. A tab no step's action opened (a page timer, a popup long after its
|
|
265
|
+
click) is not followed.
|
|
254
266
|
|
|
255
267
|
## Several people: `As "name"` (browser sessions)
|
|
256
268
|
|
|
@@ -322,6 +334,22 @@ Scenario: ...
|
|
|
322
334
|
tags means `or`; once `and`, `not` or a parenthesis appears, join every
|
|
323
335
|
tag with `and` or `or`. The `@` may be left out.
|
|
324
336
|
|
|
337
|
+
Tags that change how a scenario runs:
|
|
338
|
+
|
|
339
|
+
| Tag | Effect |
|
|
340
|
+
|-----|--------|
|
|
341
|
+
| `@api` | An HTTP API test, with no browser ([API tests](api.md)) |
|
|
342
|
+
| `@side-effects` | It changes the app's data; `run_plan` counts how often it runs |
|
|
343
|
+
| `@serial` | Never runs at the same time as another `@serial` scenario |
|
|
344
|
+
| `@serial:<name>` | Never runs at the same time as another scenario with that name (`@serial:rooms`); any case, and commas give several (`@serial:rooms,users`) |
|
|
345
|
+
|
|
346
|
+
`@serial` holds across workers, the record and heal phases, a heal's proof
|
|
347
|
+
replays of an Outline's other rows, and `--shard` (one name's scenarios go to
|
|
348
|
+
one machine). `saffron eject` writes `workers: 1` into the Playwright config it
|
|
349
|
+
creates for such a suite. Tag the scenarios that share data the app keeps
|
|
350
|
+
(one account, one booking slot) instead of running the whole suite with
|
|
351
|
+
`--workers 1`.
|
|
352
|
+
|
|
325
353
|
## A complete file
|
|
326
354
|
|
|
327
355
|
How the pieces fit in one project. The shared sign-in lives in a
|
|
@@ -393,4 +421,6 @@ What each part buys:
|
|
|
393
421
|
the recording is bound to its row, see "A recording bound to its row"
|
|
394
422
|
in HOW-TO-USE).
|
|
395
423
|
- **Each scenario ends in its `Then` block**, the strict verdict that is
|
|
396
|
-
never healed.
|
|
424
|
+
never healed. The one exception: a database check whose query fails
|
|
425
|
+
after a schema change may have its query repaired; its binds, expected
|
|
426
|
+
rows and row count never change (see [Database steps](database.md)).
|