saffron-ai 1.3.5 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "saffron-ai",
3
- "version": "1.3.5",
3
+ "version": "1.4.0",
4
4
  "description": "End-to-end tests written in plain language: an AI agent records each scenario once, replays run at zero tokens, and broken replays heal into proposals you review",
5
5
  "scripts": {
6
6
  "test": "vitest run",
@@ -47,9 +47,14 @@
47
47
  "devDependencies": {
48
48
  "@cursor/sdk": "^1.0.32",
49
49
  "@openai/codex-sdk": "^0.158.0",
50
+ "@types/mssql": "^12.3.0",
50
51
  "@types/node": "^26.1.0",
52
+ "@types/pg": "^8.23.1",
51
53
  "esbuild": "^0.28.1",
52
54
  "marked": "^18.0.5",
55
+ "mssql": "^12.7.14",
56
+ "mysql2": "^3.24.6",
57
+ "node-sql-parser": "^5.4.0",
53
58
  "typescript": "^6.0.3",
54
59
  "vitest": "^4.1.9"
55
60
  },
@@ -87,5 +92,24 @@
87
92
  "README.md",
88
93
  "CHANGELOG.md",
89
94
  "LICENSE"
90
- ]
95
+ ],
96
+ "optionalDependencies": {
97
+ "pg": "^8.23.1"
98
+ },
99
+ "peerDependencies": {
100
+ "node-sql-parser": "^5.4.0",
101
+ "mssql": "^12.7.14",
102
+ "mysql2": "^3.24.6"
103
+ },
104
+ "peerDependenciesMeta": {
105
+ "node-sql-parser": {
106
+ "optional": true
107
+ },
108
+ "mssql": {
109
+ "optional": true
110
+ },
111
+ "mysql2": {
112
+ "optional": true
113
+ }
114
+ }
91
115
  }
@@ -3,14 +3,14 @@ name: saffron
3
3
  description: Create, translate, or edit Saffron test scenarios in .saffron or .feature files, browser UI tests and HTTP API tests (@api) alike, including validating their recordings and replay. Use when asked to author or fix these test files in a Saffron project. Do not activate for ordinary application development, generic Cucumber projects, or requests only to run existing tests, inspect reports, review proposals, or configure Saffron.
4
4
  license: Saffron Free Use License v1.0 (see the saffron-ai package LICENSE)
5
5
  metadata:
6
- saffron-ai: "1.3.5"
6
+ saffron-ai: "1.4.0"
7
7
  author: saffron-ai
8
8
  homepage: https://saffron-ai.io
9
9
  ---
10
10
 
11
11
  # Saffron
12
12
 
13
- This skill describes saffron-ai 1.3.5 (see the compatibility page: https://saffron-ai.io/docs/compatibility).
13
+ This skill describes saffron-ai 1.4.0 (see the compatibility page: https://saffron-ai.io/docs/compatibility).
14
14
 
15
15
  Apply this workflow only when asked to create, translate, or edit Saffron
16
16
  scenarios in `.saffron` or `.feature` files. A Saffron mention or an
@@ -96,14 +96,17 @@ end up with something recorded against it. Put
96
96
  the final assertions **last**; the trailing block of `Then` steps is
97
97
  strict under every policy. Use `Given`/`When` for actions. A text check
98
98
  passes only on text the page shows: hidden elements, hidden parts of an
99
- element and hidden dropdown options never count, so "should be shown"
100
- means visible.
99
+ element, hidden dropdown options and frames the page hides never count,
100
+ so "should be shown" means visible. One narrow exception to "never healed": a database check
101
+ whose query fails after a schema change may have its query (table,
102
+ columns, joins) repaired by the agent; its binds, expected rows and row
103
+ count never change.
101
104
 
102
105
  ## 3. Never bake volatile or secret values into files
103
106
 
104
107
  - Secrets: `{env:VAR}`, as in `When I enter "{env:ADMIN_PASSWORD}" in the password field`.
105
108
  - Shared test data (accounts, names, enum values): put it in `data/*.json` and write `{data:users.admin.email}` (file, then dotted path; one value, not a list). The recording stores the token, so editing the file changes the next replay at zero tokens. Never for secrets: data files are committed. To check a whole list (an enum), point a Then step at it: `Then the status filter should list every {data:enums.OrderStatus}`; replay iterates the file's list, so adding a value needs no re-record. `Examples: {data:roles}` reads outline rows from `data/roles.csv` or a JSON list (`.saffron` only). A value that must be new on every run (an email for a sign-up) is `{unique:name}`: fresh per run, the same within it.
106
- - State the application must be in before the suite (seeded users, a reset database): do not script it through the UI in every scenario. The project's `saffron.config.json` takes `"setup"` and `"teardown"` shell commands, run once around `saffron run` (teardown always runs). Propose a seed script there; keep scenario steps for the behaviour under test.
109
+ - State the application must be in before the suite (seeded users, a reset database): do not script it through the UI in every scenario. The project's `saffron.config.json` takes `"setup"` and `"teardown"` shell commands, run once around `saffron run` (teardown always runs). Propose a seed script there; keep scenario steps for the behaviour under test. Data one scenario needs can be seeded inside it instead with a database seed step (`Given a customer "ann-{unique:c}@example.test" on the "basic" tier`), when the project configures `db` with writes on; see Database steps below.
107
110
  - Dates: write intent when the day should move ("1 day from today" records as `{date+1}`); a date you write out (`I enter 2026-10-10`, in a table, a data file or `{env:...}`) is entered as written on every run. An Examples cell such as `{date+2:DD.MM.YYYY}` gives each row its own day. A date the app shows in words or as a range is a token too (`{date+7:D. MMM:de-DE}`, `{date+7 - +9:MMM D, YYYY}`), never a setup script.
108
111
  - Dynamic display values: capture and compare, e.g. `I record the total as "first"` … `"first" should differ from the displayed total`.
109
112
 
@@ -296,6 +299,28 @@ or "should match the contract", and name the status of an error case.
296
299
  Pin the contract first (`npx saffron api pin <url or file>`), and read
297
300
  [API tests](references/api.md) for the recording, review and run details.
298
301
 
302
+ ## Database steps (beta)
303
+
304
+ ```gherkin
305
+ Scenario: An upgrade is saved
306
+ Given a customer "ann-{unique:c}@example.test" on the "basic" tier
307
+ When I upgrade "ann-{unique:c}@example.test" to "gold" on the account page
308
+ Then the customer "ann-{unique:c}@example.test" should be on the "gold" tier in the database
309
+ And no refund should exist in the database for "ann-{unique:c}@example.test"
310
+ ```
311
+
312
+ With `db` in `saffron.config.json` (PostgreSQL, MySQL or SQL Server), a `Then` can check
313
+ what the database holds and a `Given` or `When` can seed data, in browser and `@api`
314
+ scenarios alike. Name the database in a check ("in the database") and name
315
+ the record by a value the scenario wrote: the step's text, an Examples
316
+ column, a `{unique:...}` value, or a value an earlier step kept. A check
317
+ bound to nothing ("a customer exists") is refused while recording. A
318
+ "not in the database" check waits out `db.timeout` (3 s by default) before
319
+ it passes. Seeding needs `"writes": true` and the database's host or name
320
+ in `allow`; seed what the app gives no way to set, with `{unique:...}`
321
+ values, and the rows are undone when the scenario ends. Read
322
+ [Database steps](references/database.md) for the details.
323
+
299
324
  ## Repeated step groups → StepSets (`.saffron` files only)
300
325
 
301
326
  `.saffron` is a superset of `.feature`: rename the file, nothing else
@@ -353,7 +378,10 @@ refused (red, at zero tokens) with guidance naming the column and the
353
378
  step. That includes dates without a year and one- or two-character
354
379
  values. A blank cell is refused the same way, or, when its step types
355
380
  `<column>`, the first filled row records once more and serves the blank
356
- rows too. Keep a check that lists all rows' values out of the Outline.
381
+ rows too. A value the page sets in its own way (a phone or IBAN in
382
+ blocks, an id zero-padded inside a longer one like `INV-000104`) binds the
383
+ recording to its row the same way: give the cell the value as the page
384
+ shows it. Keep a check that lists all rows' values out of the Outline.
357
385
  Tests live in the folder named by `features` in `saffron.config.json`
358
386
  (`features/` by default).
359
387
 
@@ -392,6 +420,7 @@ Use Saffron itself first: `{date...}` tokens (ranges and locales included), capt
392
420
  | Create a fixed record (`"cust-1"`) in an API step | `{unique:name}`: every run, and the proof replay, creates its own |
393
421
  | Check an API response's status in a `When` | Name the status in a `Then` ("should answer 404") |
394
422
  | One long step line carrying an API key, ids and several quoted parameters | A short step and a 2-column data table under it, one value per row |
423
+ | A database check bound to nothing ("Then a customer exists in the database") | Name the record the scenario made: `Then the customer "ann-{unique:c}@example.test" should be saved in the database` |
395
424
 
396
425
  ## References
397
426
 
@@ -403,3 +432,6 @@ Use Saffron itself first: `{date...}` tokens (ranges and locales included), capt
403
432
  heal model, secrets setup, the API settings.
404
433
  - [API tests](references/api.md): `@api` scenarios, the pinned contract,
405
434
  writing and reviewing request steps, API calls from a browser scenario.
435
+ - [Database steps](references/database.md): checks and seeds against the
436
+ configured database, what is recorded and refused, cleanup, and the one
437
+ Then whose query may be repaired.
@@ -2,7 +2,9 @@
2
2
 
3
3
  A scenario tagged `@api` tests an HTTP API with no browser. The agent
4
4
  records each request once, every later run replays it at zero tokens, and
5
- `Then` steps are never healed, as for browser scenarios. A browser
5
+ `Then` steps are never healed, as for browser scenarios (the one
6
+ exception is a database check's query after a schema change, see
7
+ [Database steps](database.md)). A browser
6
8
  scenario can also call the API in a step; see the end of this page.
7
9
 
8
10
  ## Before writing
@@ -90,6 +92,11 @@ Feature: Customers API
90
92
  2xx is recorded only when its step names that status; otherwise the
91
93
  step fails, so a failure seen while recording never replays as the
92
94
  expected result.
95
+ - **A `Then` says what it checks.** A `Then` whose only check would be
96
+ the 2xx class, or the status its request already required, is refused
97
+ (it could never fail): name a field or "should match the contract". A
98
+ time limit ("within 2 seconds") or a poll's limit is recorded exactly as
99
+ the step states it in figures.
93
100
  - **Secrets stay `{env:VAR}` tokens**, typically in a step that names the
94
101
  credential: `Given I use the API token "{env:API_TOKEN}"`. The request
95
102
  sends the value and the recording keeps the token.
@@ -44,7 +44,7 @@ your-project/
44
44
 
45
45
  | Key | Meaning |
46
46
  |---|---|
47
- | `dataDir` | Folder of JSON files read by `{data:file.key}` tokens (default `data`). Committed; not for secrets |
47
+ | `dataDir` | Folder of JSON and CSV files read by `{data:file.key}` tokens and by `Examples: {data:file}` (default `data`). Committed; not for secrets |
48
48
  | `baseURL` | App under test; steps say "the login page", not full URLs |
49
49
  | `storageState` | Playwright storage-state JSON so replays and the agent start authenticated (`npx playwright open --save-storage=.auth/state.json <url>`) |
50
50
  | `sessions` | Per browser session (`As "buyer" ...`): `{ "buyer": { "storageState": ".auth/buyer.json" } }` starts that session signed in as its own person; others, and the default session, take `storageState` |
@@ -72,7 +72,8 @@ your-project/
72
72
  | `healModel` | Cheaper model for heal sessions only |
73
73
  | `antigravityApiKey` | `true`: Antigravity signs in with `GEMINI_API_KEY` even beside an agy login (default: a login wins) |
74
74
  | `ci` | Where `saffron runs` and `import --run` find CI runs: `{"provider": "github" \| "azure", "organization": ..., "project": ...}`. Unset, an Azure Repos remote means Azure Pipelines (through `az`), anything else GitHub Actions (through `gh`). `organization` and `project` are for a GitHub repository built by Azure Pipelines. `cacheRuns` and `cacheMB` cap the runs viewed into `.saffron/ci/` (default 10 and 1024) |
75
- | `api` | The API under test: `{"baseURL": ..., "contract": "api/openapi.json", "transcript": "on-failure"}`. `baseURL` is where API steps' relative paths go (unset: the run's `baseURL`); `contract` is the pinned OpenAPI 3 document schema checks validate against (`saffron api pin` writes it), and a change to it, or to which file it names, makes the recordings checked against it stale; `transcript` keeps each scenario's requests and responses for the report and `saffron trace`: `"on-failure"` (default), `"on"` or `"off"` (statuses and timings only); `services` names more APIs, each `{"baseURL": ..., "contract": ...}`, which a request names instead of a host (`--api-url name=<url>` points one elsewhere for a run). See [API tests](api.md) |
75
+ | `db` | The database that database steps check and seed (PostgreSQL, MySQL or SQL Server, by the URL's scheme; `npm install node-sql-parser` and `pg`, `mysql2` or `mssql`): `{"url": "{env:DATABASE_URL}", "writes": false, "allow": ["localhost"]}`. `url` must be an `{env:...}` reference; `writes` (default `false`) lets `Given` and `When` steps insert and update with `dbWrite`; `allow` lists the hosts or database names a write may go to; `timeout` (default 3000 ms) is how long a check reads again, and a "not in the database" check must hold that long; `cleanup` is `"all"` (default: delete inserts, put updated rows back), `"inserts"` or `"none"`. A `Then` records `expectRows`: one SELECT, values as `$n` binds bound to a scenario value, checks on its rows; a check whose query fails on the schema may have its query repaired by the agent, its binds, checks and row count unchanged |
76
+ | `api` | The API under test: `{"baseURL": ..., "contract": "api/openapi.json", "transcript": "on-failure"}`. `baseURL` is where API steps' relative paths go (unset: the run's `baseURL`); `contract` is the pinned OpenAPI 3 document schema checks validate against (`saffron api pin` writes it), and a change to it, or to which file it names, makes the recordings checked against it stale; `transcript` keeps each scenario's requests and responses for the report and `saffron trace`: `"on-failure"` (default), `"on"` or `"off"` (statuses and timings only); `services` names more APIs, each `{"baseURL": ..., "contract": ...}`, which a request names instead of a host (`--api-url name=<url>` points one elsewhere for a run). A key Saffron does not read under `api` or a service is refused. See [API tests](api.md) |
76
77
 
77
78
  ## CLI
78
79
 
@@ -90,7 +91,7 @@ your-project/
90
91
  | `saffron mcp` / `saffron lsp` | MCP tools (`search_steps`, `list_step_sets`, `project_status`, `run_plan`, `api_contract`, `verify`) for AI assistants / language server for editors |
91
92
  | `saffron api pin <source> [--out file] [--service name]` | Pin the API contract: fetch an OpenAPI 3 document from a Swagger UI page, a document URL or a file, and write it as JSON (default `api/openapi.json`, or `api/<name>.json` with `--service`). Add it to the config as `api.contract`, or as `api.services.<name>.contract` |
92
93
  | `saffron init [--examples] [--agents list]` | Install this skill into the project's agent directories, register MCP, add an AGENTS.md block, scaffold config; `--examples` adds the Saucedemo example suite |
93
- | `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs and 1 GB unless `ci.cacheRuns` and `ci.cacheMB` say otherwise), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
94
+ | `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line; it exits 1 when the merged run has a red (or a pending review with `strict` on). `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs and 1 GB unless `ci.cacheRuns` and `ci.cacheMB` say otherwise), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
94
95
  | `saffron dashboard [--stdout] [--no-open]` | The suite's quality from the run history (failing and for how long, flaky, repeatedly healed, never run, slowest, trends, per tag); writes and opens `.saffron/reports/dashboard.html`, `--stdout` prints it |
95
96
  | `saffron export [-o file] [--branch name [--base branch]] [--report-only]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). `--report-only` writes a summary instead (report and screenshots only, `saffron-summary.zip`): upload it as a `saffron-summary*` artifact on every run so any run's results can be viewed; a run that left only summaries has nothing to import. In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
96
97
  | `saffron runs [--branch b] [--limit n] [--provider github\|azure] [--json]` | The branch's CI runs, with how many of each run's proposals are pending here; exit 2 when `gh` or `az` is missing or signed out |
@@ -0,0 +1,164 @@
1
+ # Database steps (beta)
2
+
3
+ A step can check what the database holds, or seed the data a scenario
4
+ starts from, with no query written by hand. While recording, the agent
5
+ reads the schema, writes one parameterized statement, runs it, and records
6
+ it with the step's values as binds; every later run replays it at zero
7
+ tokens. They work in browser scenarios and in `@api` scenarios alike.
8
+ PostgreSQL, MySQL and SQL Server, named by the URL's scheme
9
+ (`postgres://`, `mysql://`, `mssql://`).
10
+
11
+ ## Setup
12
+
13
+ ```json
14
+ {
15
+ "db": { "url": "{env:DATABASE_URL}", "writes": false, "allow": ["localhost", "shop_test"], "timeout": 3000, "cleanup": "all" }
16
+ }
17
+ ```
18
+
19
+ - `npm install node-sql-parser` and the engine's driver (`pg`, `mysql2` or
20
+ `mssql`) in the project. Both load only when `db` is set.
21
+ - Every engine takes `$1`, `$2` binds; write the engine's own SQL
22
+ otherwise. On MySQL and SQL Server an `INSERT` with an auto-numbered key
23
+ writes one row (several rows give the key as binds), and an `UPDATE`
24
+ never sets the key: Saffron reads the rows back by it. A recording is
25
+ refused on another engine.
26
+ - `url` must be an `{env:...}` reference: the connection string and its
27
+ password never sit in a file, and never reach a recording, a proposal,
28
+ a report, a trace or the agent. Write the host and the database in the
29
+ URL itself; a `?host=` parameter, or no host, is refused for writes.
30
+ - `writes` (default `false`) lets `Given` and `When` steps seed data.
31
+ - `allow` lists the hosts or database names a write may go to. A write
32
+ anywhere else is refused before it runs.
33
+ - `timeout` (milliseconds, default 3000) is how long a check reads again.
34
+ - `cleanup` (default `"all"`) is what is undone when a scenario ends.
35
+ - Give Saffron its own database user on a test database.
36
+
37
+ ## Wording the steps
38
+
39
+ ```gherkin
40
+ Scenario: A new customer is saved
41
+ Given a customer "ann-{unique:c}@example.test" on the "basic" tier
42
+ When I upgrade "ann-{unique:c}@example.test" to "gold" on the account page
43
+ Then the customer "ann-{unique:c}@example.test" should be on the "gold" tier in the database
44
+ And no refund should exist in the database for "ann-{unique:c}@example.test"
45
+ ```
46
+
47
+ - Name the database in a check (`in the database`), and name the record
48
+ by a value the scenario wrote: the step's own text, an Examples column,
49
+ a `{unique:...}` value, or a value an earlier step kept. A check that
50
+ names nothing ("a customer exists") would pass on rows any earlier run
51
+ left behind, and is refused while recording. A value spelled in the
52
+ step is the same every run: where the check must prove this run saved
53
+ the row, use a `{unique:...}` value or an id an earlier step kept.
54
+ - Seed with `{unique:...}` values where a column is unique: parallel
55
+ workers and proof replays each write their own.
56
+ - A seed `Given` states the data, not how to insert it. Seed only what the
57
+ app gives no way to set; a `When` that the app performs goes through the
58
+ app.
59
+ - A `Fixture:` block does not write to the database: seed in each
60
+ scenario's `Given`.
61
+
62
+ ## What is recorded
63
+
64
+ | Action | Step | Holds |
65
+ |---|---|---|
66
+ | `dbWrite` | `Given`, `When` | One `INSERT`, or one `UPDATE` with a `WHERE`; values as `$1`, `$2` binds (`body`); `extract` keeps a field of the first row written as `{captured:name}` |
67
+ | `expectRows` | `Then` | One `SELECT`; values as binds; `json` checks on its rows read as a JSON list (`count`, `/0/column` equals, `item`); `rowCount`, the rows it found when recorded |
68
+
69
+ - Values are binds, never written into the SQL: a literal is refused, and
70
+ so is a comment. Each bind is compared with a column (`email = $1`); a
71
+ query that returns a bind, compares values with no column, or adds
72
+ `OR true` is refused. The `WHERE` must tie its rows to a bind with `=`,
73
+ `IN`, `LIKE` (or `ILIKE` on PostgreSQL). Each side of an `OR`, and
74
+ every value of an `IN`, needs such a bind (`email = $1 OR role =
75
+ 'admin'` is refused); one side of an `AND` is enough. `<>`, `NOT (...)`, a
76
+ range alone, a bind in `LIMIT`, and `UNION` are refused, and a subquery
77
+ must tie its own rows. A `LIKE` value made only of `%` or `_` (an
78
+ Examples cell too) matches every row, so it is refused unless another
79
+ condition ties the rows. A JSON field can tie a check
80
+ (`data->>'orderRef' = $1`). Cast with `CAST($1 AS type)`, not `::`. A
81
+ `WITH` that selects a bind as a column is refused like a `SELECT $1`.
82
+ On MySQL join texts with `CONCAT()`: `||` runs as `OR` there and is
83
+ refused.
84
+ Binds are written as their tokens, so each Examples row checks its own
85
+ value. Bind a row's whole value (`<email>`), never a part of it ("ann"
86
+ of ann@example.test): a part is kept as written for every row and is
87
+ refused, unless every row holds it alike. An aggregate with no `GROUP BY` is always one row: check its
88
+ value (`/0/n equals 1`, or a count on a list it holds), never a count
89
+ or an exists on the row itself, also when the aggregate is in the
90
+ query's only `FROM` or `WITH` source. A count above 0, or a check that
91
+ there is no second row, under `LIMIT`, `TOP`, `FETCH`, `DISTINCT` or a
92
+ `GROUP BY` with no aggregate is refused, also when they sit in a query
93
+ read `FROM` or `WITH`, and so is any check under `OFFSET`, and `LIMIT
94
+ 0`, `TOP 0` and `FETCH ... 0`. A field that must not be in a row
95
+ (`/0/data/discount`) holds only when that row is there.
96
+ - A check reads in a transaction that is rolled back (read-only on
97
+ PostgreSQL and MySQL). A date, a timestamp, a bigint or a numeric reads
98
+ as text in UTC (`2026-11-07`, `2026-11-07 09:00:00+00` on PostgreSQL),
99
+ the same on every machine; SQL Server reads a DECIMAL as a number.
100
+ - A presence check passes as soon as its rows are there, within
101
+ `db.timeout`. A "not in the database" check (a count of 0, a `count(*)` checked as 0, a field that
102
+ must not exist) waits for a row a worker deletes late, then must hold to
103
+ the end of the window, so a row a worker writes late fails it. A check holding both waits for the first part, then holds
104
+ the second to the end.
105
+ - The run report shows each check's SQL, row count and first five rows,
106
+ masked.
107
+
108
+ ## Refused
109
+
110
+ `DELETE`, `TRUNCATE`, `DROP`, `ALTER`, `GRANT`, several statements in one,
111
+ a write inside a `WITH`, an `UPDATE` without a `WHERE`, `UPDATE ... FROM`,
112
+ `INSERT ... ON CONFLICT` and `ON DUPLICATE KEY UPDATE`, MySQL's
113
+ `INSERT IGNORE`, an `UPDATE` with a `JOIN`, a second table, `ORDER BY` or
114
+ `LIMIT`, a write to a table of another database (`otherdb.t`,
115
+ `db.schema.t`), table hints (`WITH (NOLOCK)`, `READPAST`),
116
+ `SELECT ... INTO`, MySQL's `/*! ... */` comments, a backslash before a
117
+ quote on SQL Server, a write in a `Then`, a write through a view or synonym, a MySQL read
118
+ that locks rows (`FOR UPDATE`, `FOR SHARE`, `LOCK IN SHARE MODE`), an `UPDATE` whose `WHERE`
119
+ binds nothing the scenario wrote, and any statement calling a function
120
+ whose effects reach past its transaction (`pg_terminate_backend`,
121
+ `dblink`, `set_config`, sequences, advisory locks, large objects, files, `SLEEP`, `BENCHMARK`, `LOAD_FILE`, `OPENQUERY`,
122
+ `OPENROWSET`).
123
+ Statements are judged on their parsed form, never their text.
124
+
125
+ ## Cleanup
126
+
127
+ When a scenario ends, green or red (and at Ctrl-C or SIGTERM), Saffron undoes its
128
+ writes, the latest first, by primary key and nothing else:
129
+
130
+ | `cleanup` | Inserts | Updates |
131
+ |---|---|---|
132
+ | `"all"` (default) | deleted | changed columns put back as they were before the step |
133
+ | `"inserts"` | deleted | left |
134
+ | `"none"` | left | left, for debugging |
135
+
136
+ A write that must be undone needs a table with a primary key; an `UPDATE`
137
+ that changes a primary key is refused under `"all"`. What a recording
138
+ wrote is undone before its proof replays it, and what the proof wrote
139
+ before the stability and row proofs. A row cleanup cannot undo is named
140
+ on the console. A killed run (SIGKILL, out of memory) undoes nothing, and
141
+ rows a database trigger writes are never undone: sweep them in
142
+ `teardown`. `run --plan` lists a write that `cleanup` keeps as a side
143
+ effect.
144
+
145
+ ## The one Then that may be repaired
146
+
147
+ A `Then` is never healed, with one exception. When a database check's
148
+ stored query fails because the schema changed (a table or column gone, a
149
+ type that no longer fits), the agent repairs where the query looks: its
150
+ table, columns and joins. Its binds, its expected rows and its row count
151
+ are locked: a repair that changes any of them, or adds a `LIMIT`,
152
+ `DISTINCT`, `GROUP BY` or aggregate, is refused, and the step stays red.
153
+ A check that expects no rows, a `count(*)` of 0, a count from an Outline
154
+ column, or a field that must not be there, is not repaired: record it again.
155
+ A repaired check, and a heal that adds or changes a database write, is
156
+ left out of `accept --all`: it is accepted by file after review. The repaired check must hold on the live data and on a proof
157
+ replay; the step is reported healed and a proposal names the old and new
158
+ SQL. A query that runs and finds other rows is plain red, and no agent is
159
+ started.
160
+
161
+ ## Not yet
162
+
163
+ MongoDB; `saffron eject` writes these scenarios as `test.fixme`; the
164
+ editor plugins show database steps only as the runner reports them.
@@ -48,6 +48,13 @@ Scenario Outline: Failed login shows an error
48
48
  | standard | wrong | Wrong username or password |
49
49
  ```
50
50
 
51
+ A `<name>` within an edit or two of a column (a misspelt `<rol>` beside
52
+ `| role |`) is a parse error naming the columns; other tags, such as
53
+ `"<script>"` or `List<String>`, stay literal. An Outline with no Examples
54
+ whose steps use a placeholder is refused (add Examples or use
55
+ `Scenario:`), and so is an Examples block with no table
56
+ (`Examples: data:roles`; the file form is `Examples: {data:roles}`).
57
+
51
58
  ## Data tables
52
59
 
53
60
  **2-column = key/value.** Values record as `<table:key>` references:
@@ -171,7 +178,8 @@ is 9 digits.
171
178
  `{env:...}`) is entered as written on every run. In an Outline, each
172
179
  Examples row's `{date+N}` cell, in any format, gives that row its own day.
173
180
  - Text assertions read only what the page shows: hidden text, hidden
174
- options and hidden parts of an element never satisfy them.
181
+ options, hidden parts of an element and anything inside a frame the
182
+ page hides never satisfy them.
175
183
  - Capture and compare displayed values instead of literals:
176
184
 
177
185
  ```gherkin
@@ -251,6 +259,10 @@ When I open a new tab at the admin page
251
259
  Recorded as `switchTab` / `closeTab` / `newTab` with 0-based tab indexes
252
260
  in creation order; replay repeats them in order. Say which tab in the
253
261
  step ("the first tab", "the preview tab") so the recording is unambiguous.
262
+ When the steps after the click keep working on the page clicked, the
263
+ recording switches back to it, so replay runs them there, not on the new
264
+ tab. A tab no step's action opened (a page timer, a popup long after its
265
+ click) is not followed.
254
266
 
255
267
  ## Several people: `As "name"` (browser sessions)
256
268
 
@@ -409,4 +421,6 @@ What each part buys:
409
421
  the recording is bound to its row, see "A recording bound to its row"
410
422
  in HOW-TO-USE).
411
423
  - **Each scenario ends in its `Then` block**, the strict verdict that is
412
- never healed.
424
+ never healed. The one exception: a database check whose query fails
425
+ after a schema change may have its query repaired; its binds, expected
426
+ rows and row count never change (see [Database steps](database.md)).