saffron-ai 1.3.5 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,655 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
5
5
  JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
6
6
  release heading names its main feature: `## <version>: <title>`.
7
7
 
8
+ ## 1.4.0: Database steps
9
+
10
+ Released 2026-11-10.
11
+
12
+ Runner
13
+
14
+ - Database steps (beta): a step like `Then the customer should
15
+ be saved in the database` works with no query written by hand. While
16
+ recording, the agent reads the schema and records one parameterized
17
+ `SELECT`, its values as binds; every later run replays it at zero
18
+ tokens. A check must be bound to a value the scenario wrote; a value that
19
+ is new on each run (`{unique:...}`, an id an earlier step kept) keeps it
20
+ from passing on rows an earlier run left. With `"writes": true` and the
21
+ database's host or name in `allow`, a `Given` or `When` may seed data with
22
+ one `INSERT` or `UPDATE ... WHERE`. When the scenario ends, green or
23
+ red, `db.cleanup` undoes them: `"all"` (default) deletes the rows it
24
+ inserted and puts the rows it updated back as they were, `"inserts"`
25
+ deletes inserts only, `"none"` leaves everything. `DELETE`, `TRUNCATE`,
26
+ DDL and several statements are refused by a SQL parser. Set `db` in
27
+ `saffron.config.json` and `npm install node-sql-parser` with the
28
+ engine's driver: neither comes with saffron-ai, so a project without
29
+ database steps does not carry the parser's 88 MB. Recordings with
30
+ database steps are cache version 20.
31
+ - Database steps on PostgreSQL, MySQL and SQL Server, named by db.url's
32
+ scheme (`postgres://`, `mysql://`, `mssql://`), with the driver `pg`,
33
+ `mysql2` or `mssql`. Each engine takes the same steps, rules and `$1` binds; MySQL and SQL Server writes
34
+ are read back by their primary key, one row per `INSERT` when the key
35
+ is auto-numbered. A recording made on one engine is refused on another.
36
+ A write to more than one table (MySQL's `UPDATE a JOIN b`) is refused,
37
+ and a write whose rows cannot be read back by their key is rolled back,
38
+ so cleanup never misses a row or touches one Saffron did not write. A
39
+ SQL Server check that meets a row the app is still writing reads again
40
+ instead of failing. A table hint (`WITH (NOLOCK)`, `READPAST`) is refused,
41
+ so a check never reads rows the app did not commit or skips rows it
42
+ holds. A write goes only to the database `db.url` names (MySQL's
43
+ `otherdb.t` and SQL Server's `db.schema.t` are refused), an `UPDATE` with
44
+ `ORDER BY` or `LIMIT` is refused, and a SQL Server `UPDATE` that changes a
45
+ time finer than milliseconds, or with an offset, is refused while cleanup
46
+ would put it back. MySQL's `INSERT IGNORE` is refused, as it skips rows
47
+ without saying so, and a check whose rows the app holds locked for the
48
+ whole `db.timeout` fails saying so.
49
+ - A database check reads again within `db.timeout` (default 3000 ms): a
50
+ presence check passes as soon as its rows are there, and a "not in the
51
+ database" check waits for a row a worker deletes late, then must hold
52
+ to the end of the window, so a row a worker writes late fails it.
53
+ - On PostgreSQL and MySQL every statement a database step runs goes to
54
+ the database as exactly one statement, so a query the SQL check reads
55
+ as one SELECT can never run a second statement (a quote read differently
56
+ by the two could hide a COMMIT and a DELETE in a look at the data). SQL
57
+ Server runs a batch, so there the parser's one-statement rule and the
58
+ refused backslash before a quote hold it.
59
+ - Cleanup puts a json or jsonb column back as JSON, an array or a string
60
+ value included. Placeholders numbered with a gap ($1 and $3, no $2) are
61
+ refused while recording.
62
+ - Only a scenario whose recording has database steps, or whose steps name
63
+ the database before it is recorded, needs the database to answer before
64
+ recording starts.
65
+ - A database check that the database refused (a permission, a timeout)
66
+ says so, apart from one that found other rows; neither starts an agent.
67
+ - A database check whose stored query fails because the schema changed
68
+ is repaired by the agent, the one exception to a `Then` never being
69
+ healed: only its table, columns and joins may change, its binds and
70
+ expected rows are enforced unchanged, it must return as many rows as
71
+ the recording did (a join that fans out is refused), and the repair must
72
+ pass a proof replay. The step is reported healed and a proposal names the old and
73
+ new SQL. A query that returns the wrong rows stays red, with no agent.
74
+ - The run report shows the rows each database check found under its
75
+ step, masked, the first five.
76
+ - A recorded check must tie its rows to the scenario's value: a column
77
+ compared with a bind by `=`, `IN` or `LIKE`, on each side of an `OR`,
78
+ and a subquery on the tying side ties its own rows the same way.
79
+ `UNION`, `INTERSECT` and `EXCEPT`, `email <> $1`, `NOT (...)`, a
80
+ comparison inside `COALESCE` or `CASE`, a range alone
81
+ (`created_at > $1`) and a bind in `LIMIT` or `OFFSET` are refused, as
82
+ each could hold on rows the scenario never wrote. The same holds for an
83
+ `UPDATE`'s `WHERE`. A JSON field can tie a check
84
+ (`data->>'orderRef' = $1`, `data @> CAST($1 AS jsonb)`). A null bind,
85
+ or an empty one in a "not in the database" check, is refused while
86
+ recording and at replay: `= NULL` matches no row.
87
+ - A repaired database check, and a heal that adds or changes a database
88
+ write, are held for review: `accept --all` leaves them out (`DATABASE
89
+ CHECK REPAIRED`, `DATABASE WRITE ADDED`).
90
+ - A database that is down, restarts mid-step or does not answer within 10
91
+ seconds is the environment: no agent session starts, and the message
92
+ says the database did not answer, not the app. Before recording, an
93
+ unset `db.url` variable, a connection string that is not a URL, a wrong
94
+ password, a missing database and a missing driver (`pg`, `mysql2` or
95
+ `mssql`) or `node-sql-parser` are said at once, without starting an
96
+ agent.
97
+ - `saffron status --json` and the `project_status` MCP tool say whether
98
+ database steps are configured (`database`: the engine, whether writes
99
+ are on, the cleanup mode), never the connection string; they and the
100
+ editors' config views mask a password in a `db.url` written in keyword
101
+ form (`host=... password=...`) or in its query (`?password=`). The
102
+ bundled skill, `saffron init`'s AGENTS.md block and the guides cover
103
+ database steps and the one Then whose query may be repaired.
104
+ - `run --plan` and the MCP `run_plan` tool list a recorded database write
105
+ as a side effect when `db.cleanup` keeps it (`"inserts"` keeps an
106
+ `UPDATE`, `"none"` every write).
107
+ - An `@api` scenario whose recording sends no request (database steps
108
+ alone) replays without the API answering.
109
+ - The language server (JetBrains completion and hover) says a step whose
110
+ recordings all read or write the database replays at zero tokens only in
111
+ the scenario that recorded it, per `As "..."` session, as VS Code does,
112
+ and its similar-wording hint no longer suggests such a step.
113
+ - Found in the release review, before release:
114
+ - A write inside a `WITH` passed when both statements were `UPDATE`s,
115
+ and its table was never put back. Any `WITH` in a write is refused,
116
+ and so is a `WHERE` that reads no column (`WHERE true`).
117
+ - A check could echo its bind back (`SELECT $1 AS email`), or hold on
118
+ any row (`OR true`, `OR email IS NOT NULL`, `(email = $1) IS NOT NULL`,
119
+ `$1 = $1`), and a schema repair could add
120
+ `LIMIT 1`, `DISTINCT` or `max()` to hide a join that fans out. Both
121
+ are refused, and so are values written in as `E'...'`, `$$...$$` or
122
+ `DATE '...'`.
123
+ - Under `"assertionPolicy": "adaptable-mid"`, a heal rewrote a database
124
+ check in the middle of a scenario, its binds and expected rows too.
125
+ A database check is never healed, under every policy.
126
+ - A database check recorded by another scenario was reused with that
127
+ scenario's value, so it could pass on rows an earlier run left. A
128
+ database step is seeded only from the scenario's own recording.
129
+ - A `db.url` written as `host=... password=...` was shown whole in the
130
+ error, password included. The message no longer shows the value, and
131
+ the rows an error shows are masked before they are cut.
132
+ - A database connection dropped mid-step crashed the run, a host that
133
+ does not answer now fails within 10 seconds, and a step still running
134
+ after Ctrl-C opens no new connection.
135
+ - Ctrl-C during a database write left its row behind. Closing now
136
+ waits for the write and undoes it.
137
+ - A check on an aggregate's one row passed whatever the table held:
138
+ `count(*)` with no `GROUP BY`, an aggregate the parser reads as a
139
+ plain function (`json_agg`, `JSON_ARRAYAGG`, `bool_or`), `GROUP BY
140
+ ROLLUP`, and a query whose only `FROM` or `WITH` source is an
141
+ aggregate. A count, exists or item check on that row is refused while
142
+ recording and at replay; only a value checked on it is taken.
143
+ - A count above 0, or a check that there is no second row, could not see
144
+ a second row under `LIMIT`, `TOP`, `DISTINCT` or a `GROUP BY` with no
145
+ aggregate, and `OFFSET` (SQL Server's `OFFSET ... FETCH` too) hid rows
146
+ from a count and let a "no rows" check pass: each is refused. On MySQL
147
+ and SQL Server a check never saw `DISTINCT` (the parser writes it
148
+ differently there), so such checks passed on a duplicate row and a
149
+ repair could add it; it is now read on every engine.
150
+ - A check that expects no rows, or that a field must not be there, is no
151
+ longer repaired: a repaired query finding none proved nothing, and a
152
+ repair could stop selecting that field.
153
+ - A field that must not be in a row (`/0/data/discount`) passed when the
154
+ app saved no row at all; it now needs the row to be there.
155
+ - A captured id of one or two characters was recorded as itself in a
156
+ database bind, so replay checked the recording run's row; it must be
157
+ written as `{captured:name}`, and one written as itself beside another
158
+ bind is refused.
159
+ - On MySQL, `--` with no space after it let a look at the data run what
160
+ Saffron read as a comment; it is refused. On SQL Server, the agent's
161
+ `UPDATE` was read as PostgreSQL when checking that its `WHERE` names
162
+ the scenario's rows, and a write to a table whose key the driver reads
163
+ inexactly (a decimal past 15 digits) is refused while cleanup is on.
164
+ - A page fingerprint left by a browser tool that failed went onto the
165
+ next database or HTTP action, and that recording was refused on every
166
+ load after.
167
+ - A MySQL write could read back a row that was there before (`7`
168
+ finding `'07'`) for one its key missed, and cleanup deleted it: each
169
+ key now reads back exactly its own row, or the write is rolled back.
170
+ A JSON column is put back as the text it held, never read through a
171
+ JS number.
172
+ - A secret a driver quotes cut short (MySQL's duplicate entry at 64
173
+ characters, SQL Server's truncated value), one holding a parenthesis
174
+ or cut to 4 characters too, is masked, and a secret in a row no longer
175
+ breaks the rows the report shows. A lock held for the whole window
176
+ says it timed out, not that it found other rows.
177
+ - A database check was seeded from the scenario's own recording after
178
+ an earlier step changed the value it binds; it is recorded again. A
179
+ stale recording whose edited steps name the database now asks the
180
+ database to answer before recording.
181
+ - A SQL Server `BIGINT` identity past 2^53 is read exactly, so cleanup
182
+ deletes the scenario's row, never the app's.
183
+ - A check or an agent `UPDATE` with an `OR` whose other side is a fixed
184
+ value (`email = $1 OR role = 'admin'`) passed on rows earlier runs
185
+ left: each side of an `OR`, and every value of an `IN`, must hold a
186
+ value the scenario wrote. A `LIKE` value of only `%` or `_` (from an
187
+ Examples cell too) is refused unless another condition ties the rows,
188
+ and so are `LIMIT 0`, `TOP 0` and `FETCH ... 0`.
189
+ - A quoted, qualified column (`` `o`.`total` ``, `"o"."total"`) was
190
+ refused as a value written into the SQL; it is read as a name.
191
+ - Stability proofs and an Outline's row proofs failed on rows the first
192
+ proof seeded: its database writes are now undone before they run.
193
+ - The MCP `search_steps` tool showed the agent a database step's
194
+ session-prefixed text (`As "name" ...`), which would move that step
195
+ into the named session; it no longer does.
196
+ - An Outline check that holds a row's value as the page sets it in type
197
+ (a curly apostrophe or quotes, spaces collapsed or no-break, an
198
+ ellipsis) let another row pass on it; that row now records again or is
199
+ refused.
200
+ - A key under `api` or `api.services.<name>` that Saffron does not read
201
+ (`"transcripts"`, `"contracts"`) is refused with the keys it takes,
202
+ instead of leaving that setting at its default.
203
+ - A check that something is shown (text, visible, a field's value) no
204
+ longer passes on content inside a frame the page hides (`visibility:
205
+ hidden`, zero-sized, placed off the page), or inside a frame in one.
206
+ - A base URL's password, which Chromium keeps in the page's URL, is
207
+ masked in what the recording agent reads: snapshots, tool errors,
208
+ Saffron tool replies and the heal prompt's live page and failure.
209
+ Navigating to the masked URL still signs in.
210
+ - A remote browser's endpoint (`transport.browser`) no longer prints its
211
+ password or `?token=` key when the connection fails.
212
+ - `saffron status --json`, the `project_status` and `api_contract` MCP
213
+ tools mask a password in a base URL and a key in a URL's query or
214
+ fragment (`?token=`, `?api_key=`, `?key=`, `#access_token=`); `api_contract` showed the password
215
+ to the agent. The MCP `verify` tool masks them in a failed run's last
216
+ lines, and `saffron api pin` in what it prints.
217
+ - `saffron api pin` says why a URL could not be fetched (connection
218
+ refused, host not found) instead of only "fetch failed".
219
+ - `saffron run --json` without `--plan` exits 2 instead of starting a run
220
+ that prints no JSON.
221
+ - `saffron eject --out <dir>` warns when the project's Playwright config
222
+ looks for tests in another folder, so `npx playwright test` would not
223
+ run the new specs.
224
+ - `saffron init` registers the MCP server as `npx -y -p saffron-ai saffron
225
+ mcp`: a bare `npx saffron` fetched npm's unrelated `saffron` package in
226
+ a project without saffron-ai installed.
227
+ - A `saffron.config.json` that is not valid JSON is named in the error.
228
+ - `saffron status --json` and the `project_status` MCP tool name, in each
229
+ `lastRun.attention[]` entry, why a red was not run (`blocked`:
230
+ `recorder`, `environment`, `provider` or `budget`), so editors list it
231
+ as not run instead of failed.
232
+ - `--project-root` naming a folder that does not exist stops with exit 2,
233
+ instead of reading it as an empty project, and `saffron run` names the
234
+ configured features folder when it does not exist.
235
+ - A run notes when the selected data environment (`--env`, `SAFFRON_ENV`
236
+ or `env`) matches no data file, so a misspelled name no longer uses the
237
+ base data in silence.
238
+ - `Examples: {data:...}` refuses a CSV file that names a column twice, as
239
+ an Examples table does; one of them was read for both. A data JSON file
240
+ saved with a byte order mark is read.
241
+ - The recording agent can no longer send an `{env:...}` value to another
242
+ site by writing the site as a reference (`{captured:link}?p={env:...}`,
243
+ a row's `<column>`, a `{data:...}` value): the site is checked as the
244
+ browser will open it.
245
+ - A capture, and a check that a value changed, wait while the element
246
+ they were recorded on is still empty, instead of reading the same test
247
+ id in an iframe or a fallback that happens to show text: "the price
248
+ changed" passed on an embedded widget's copy.
249
+ - When a tab closes (a step closes it, or it closes itself, as a sign-in
250
+ popup does), replay and an exported Playwright test go where the
251
+ recording's browser went: they stay where they were when another tab
252
+ closed, and take the tab that slid into the closed one's place, never
253
+ simply the last tab.
254
+ - `saffron eject` no longer writes a base URL's password, or a key in its
255
+ query or fragment (`?api_key=`, `#access_token=`), into
256
+ `playwright.config.ts` or `saffron-helpers.ts`: it writes `***` and
257
+ warns, naming the variable to set to the full URL (`BASE_URL`,
258
+ `API_URL` or `API_URL_<NAME>`).
259
+ - When a click opens a new tab and the agent keeps working on the page it
260
+ clicked (the browser it records with stays there), the recording now
261
+ switches back to that page, so replay runs those steps there instead of
262
+ on the new tab, where a "not shown" check could pass on the wrong page.
263
+ A tab closed after such a click is closed from the agent's page too, and
264
+ a heal that takes over on a tab replay followed records a switch to the
265
+ tab the agent works on (its browser starts on the first tab). A
266
+ recording made earlier keeps its behavior: re-record a scenario that
267
+ opens a tab and then acts on the page that opened it.
268
+ - Replay follows only a tab a recorded action opened. One that opens
269
+ without such an action (a popup more than 5 seconds after its click, a
270
+ page timer) is not followed, as the agent's browser stayed on its page:
271
+ the checks after it ran on the new tab, where they could pass. Exported
272
+ tests follow tabs the same way.
273
+ - An older recording that uses tabs without tab marks still follows any
274
+ new tab, also one that opens later than 150 ms after a click: the check
275
+ after it moves to the new tab, and an action waits for the tab or its
276
+ element, a `{unique:...}` target included. A click that opens no tab is
277
+ not slowed. A "not shown" check there must hold on the page it started
278
+ on, then waits for the tab and must hold on it too, so neither page
279
+ passes it for the other (record the scenario again to add tab marks and
280
+ drop the wait).
281
+ - Exported tests wait for a tab a background click opened, and take a tab
282
+ that already opened before a switch or close, as replay does.
283
+ - Reports and run history mask a key in the base URL's query or fragment
284
+ (`?api_key=`, `#access_token=`), as they already masked its password.
285
+ - A signed storage URL's signature (an Azure SAS `?sig=`) is masked
286
+ wherever a key in a URL's query is.
287
+ - `saffron eject` warns when the exported tests send API requests while
288
+ `transport.api` or `transport.module` is set: they go without its
289
+ headers, proxy or certificate setting.
290
+ - A feature path a Windows machine recorded (`features\login.feature`)
291
+ is found on macOS and Linux: `saffron prune` deleted such a recording of
292
+ a live feature outside the features folder, `saffron import` refused a
293
+ Windows CI run's proposals as "not in this checkout", `saffron trace`
294
+ called them stale, `accept` could not apply their feature edits, and
295
+ viewing that run hid "Open scenario". `saffron coverage` writes feature
296
+ paths with forward slashes, and `--base` reads a map committed on
297
+ Windows as the same files instead of every scenario removed.
298
+ - Feature paths written on Windows also match elsewhere in the run: a
299
+ sharded run split across Windows and Linux machines gives every
300
+ scenario to exactly one shard (some ran twice and some never), `saffron
301
+ diff <feature file>` finds a proposal filed on Windows, `run --plan`
302
+ treats such a pending proposal as the run does instead of planning a
303
+ recording, and `status --json` names an Examples row's scenario
304
+ (`baseScenario`) from a report written on Windows. `saffron init` keeps
305
+ the Windows line endings of an AGENTS.md or CLAUDE.md that uses them.
306
+ - A dropdown option picked in a form is replayed by its label when
307
+ another option's value is the same text (that earlier option was chosen
308
+ instead). Exported tests do the same.
309
+ - Recording with many agent workers no longer prints a
310
+ MaxListenersExceededWarning: the sessions share one exit listener.
311
+ - A fixture whose pending proposal is deleted or unreadable mid-run (with
312
+ proofs off) makes its scenarios red, naming why, instead of ending the
313
+ run without a report.
314
+ - `saffron accept` writes a feature edit atomically, keeping the file's
315
+ line endings, permissions and symlink, so a crash mid-write no longer
316
+ leaves it cut short.
317
+ - One StepSet file that does not parse, or defines a set twice, no longer
318
+ stops `saffron steps` and the MCP `search_steps` and `list_step_sets`
319
+ tools: the other files' steps are offered and that file is named.
320
+ - The language server refreshes `{data:...}` hints when files in a
321
+ configured `dataDir` change, not only in `data/`.
322
+ - Recording keeps right, middle and Ctrl or Shift clicks: Playwright MCP
323
+ writes their options over several lines, and they were left out of the
324
+ recording.
325
+ - A `{unique:name:digits}` value sent as a number (in a request body or a
326
+ database check bind) is recorded as its token: it was kept as the
327
+ recording run's number, so later runs checked that run's row.
328
+ - An abbreviated amount (`€1.5M`, `$1.2K`), or one written in words or
329
+ spaced off ("3 bn", "€1.5 million", "1,5 Mio. €", "5 lakh"), is refused
330
+ instead of read as its digits, and Indian lakh grouping
331
+ (`₹1,23,456.00`) reads as 123456, in replay and exported tests.
332
+ - Copies of a request sent at once (`concurrent`) are never sent again by
333
+ `retries`: a retry could pass a race the first copies lost.
334
+ - Response bodies are read in the charset their Content-Type names, for
335
+ API requests, the browser's response checks and exported tests, and a
336
+ leading byte order mark is ignored: a Latin-1 body read garbled, so a
337
+ `differs` check could pass on an unchanged value.
338
+ - A recording the agent's provider failed (a sign-in that expired, the
339
+ network, a usage limit) is counted as blocked by the agent's provider,
340
+ not as failed in the app.
341
+ - Stopping a run (Ctrl-C, a cancelled CI job) also stops a running Codex
342
+ or Antigravity session, which kept running and spending tokens.
343
+ - `maxSessionMinutes` also stops a session that makes no tool call (a
344
+ vendor waiting on a sign-in, a stalled model call), two minutes past its
345
+ budget, and says why. Such a recording counts as failed, not as blocked
346
+ by the provider, as the app may be what kept it waiting.
347
+ - `saffron import` refuses bundles whose manifests, reports and proposals
348
+ would unzip to more than 1 GB together, before reading any, and masks a
349
+ password or token in a bundle URL wherever it prints it.
350
+ - Recording keeps the checkboxes and radio buttons the agent sets while
351
+ filling a form, a key press of `{`, `(`, `[`, `,`, `'` or `\`, and a
352
+ navigation to an address with an apostrophe; each was dropped without a
353
+ warning, and replay never did it.
354
+ - `saffron trace` refuses a trace file that unzips past its declared size
355
+ or over 1 GB, so a damaged or hostile trace from CI cannot fill memory.
356
+ - A "not shown" check passed while the text was on screen with an
357
+ invisible format character in it: the bidi isolates Fluent and ICU put
358
+ around a value, a soft hyphen, a word joiner. These now count as shown.
359
+ - A step seeded from another with a new quoted value wrote it into a
360
+ pattern unescaped: `matches "Paid (partial)"` passed on "Paid partial".
361
+ - A text check whose expected value holds a captured credential
362
+ (`{captured:...}`) is no longer handed to the page's own scripts, as
363
+ `{env:...}` values already were, so a script on the page or in an iframe
364
+ cannot read it.
365
+ - A closed single dropdown reads as its selected option only: a text,
366
+ pattern, capture or "differs" check no longer passes on another option,
367
+ and a "not shown" check fails while the dropdown shows the option it
368
+ names. A text target ("I see Cancelled") never matches an option the
369
+ dropdown only offers. A list check still reads every option.
370
+ - A "not shown" check by role and name fails when the page shows that name
371
+ in another case ("CANCEL ORDER").
372
+ - Seeding by quoted value keeps every date token form (`{date}`,
373
+ `{date+1}`, `{date+1:D MMM}`) as written in a pattern.
374
+ - A multipart request never sends `saffron.config.json` (in any letter
375
+ case), which may hold a base URL's password.
376
+ - A CI run's results never offer to open a feature path outside the
377
+ project.
378
+ - The language server reads the data environment as a run does: a
379
+ `SAFFRON_ENV` set in the shell, even empty, keeps the `.env` one out.
380
+ - A request the page started before an action (a long-poll opened on
381
+ load) is no longer taken for the action's own when its event arrives a
382
+ moment late, so "not shown", count and response checks no longer wait it
383
+ out: four such checks took 14 seconds instead of 2.
384
+ - A check that a dropdown offers an option reads the option itself
385
+ (`expectText` on `option "Cancelled"`). A 1.3.x check on the dropdown
386
+ that passed on one of its other options is now red: record it again.
387
+ - Under `--no-agent`, a run where the app or the database does not answer
388
+ marks those scenarios blocked by the environment, as a run with the
389
+ agent does, so CI tells an outage from a regression.
390
+ - Colour only on a terminal, and never with `NO_COLOR` set: a log file or
391
+ a pipe gets plain text.
392
+ - An exported "not shown" check counts as shown what replay does: a text
393
+ with invisible format characters in it, a role's name in another case,
394
+ the selected option of a closed dropdown.
395
+ - `saffron eject` says to install `@playwright/test` before
396
+ `npx playwright test`.
397
+ - A message watched for with "and see" (a toast, a flash, a status) is also
398
+ watched inside the page when its target is a test id or plain CSS: one
399
+ shown for a few milliseconds is seen, even on a busy machine or when the
400
+ page moves on at once, where a look every 50 ms could miss it. A watched
401
+ text that holds a secret is never handed to the page.
402
+ - A {unique:...} value the page shows in another case (uppercase through
403
+ CSS) goes back to its token in the recording, so a "not shown" check
404
+ never keeps last run's value; a {unique:name:digits} value the page
405
+ groups ("984,905,203") is refused when the recording is saved, as no
406
+ token stands for it.
407
+ - The check after a heal follows only a tab its action was recorded
408
+ opening, as replay does: a popup a page timer opened mid-check no longer
409
+ passes the check on another page.
410
+ - A secret in a URL's query or fragment typed on the command line
411
+ (`--base-url "https://app.test/?access_token=..."`) is masked in the
412
+ report's command, the HTML report, the console summary and export
413
+ bundles, as a URL password already was.
414
+ - A run whose only reds are scenarios the recorder could not run (a stale
415
+ or missing recording under --no-agent) no longer reads "all clear" in
416
+ the HTML report or "0 failed" alone in `saffron status`: both say the
417
+ run failed.
418
+ - saffron.config.json refuses a number or other non-string for a path or
419
+ name setting (baseURL, features, dataDir, storageState, timezone, model
420
+ and others, and a session's storageState) at load, instead of failing
421
+ later with a raw Node error after agent work.
422
+ - `saffron report --merge` exits 1 when the merged run has a red, or a
423
+ scenario pending review with `strict` on, as `saffron run` does.
424
+ - `saffron diff` with a proposal file that is not there, and `saffron
425
+ author` with a requirements file that is not there, say "<file> does not
426
+ exist" and exit 2 instead of crashing.
427
+ - A placeholder an Outline's Examples have no column for (a misspelt
428
+ `<rol>` beside `| role |`), in a step, doc string or table cell, is now
429
+ an error naming the columns (with inline Examples, only a near miss of a
430
+ column: other tags such as `"<script>"` or `List<String>` stay
431
+ literal), instead of every row running the same text and later rows
432
+ passing on row 1's recording.
433
+ - An Examples block with no table (`Examples: data:roles`, `Examples:
434
+ {roles}`, or a table forgotten) is now an error instead of running no
435
+ row without a word. A header row with no rows yet still runs nothing.
436
+ - A Scenario Outline with no Examples whose steps use a placeholder is
437
+ now refused ("add Examples or use Scenario:") instead of running once
438
+ with its placeholders as literal text.
439
+ - A `*` step opening a scenario after a Background `Given` is no longer
440
+ counted as part of the Background's setup.
441
+ - A .feature file whose description or doc string has a line starting
442
+ `Fixture:` is no longer told to rename to .saffron. Only a `Fixture:`
443
+ line with steps under it is.
444
+ - Editor snippets escape `$`, `\` and `}` in step texts and StepSet names,
445
+ so a name holding one completes as written.
446
+ - The AGENTS.md block `saffron init` writes names the features and data
447
+ folders `saffron.config.json` sets, not always `features/` and `data/`.
448
+ - `saffron init` reads a VS Code `.vscode/mcp.json` with comments and
449
+ trailing commas (JSONC) instead of calling it invalid JSON; one already
450
+ holding the saffron entry is left as is, and one that needs it says what
451
+ to add by hand rather than drop the comments.
452
+ - A database check written as count(*) and checked as 0 is now treated as
453
+ "not in the database": an empty bind is refused, the 0 must keep holding
454
+ for all of db.timeout, and a schema repair of it is refused.
455
+ - A schema repair of a database check whose row count comes from an
456
+ Outline value (a count of `<n>`) is refused, since that value may be 0.
457
+ - On MySQL, a check that locks rows (FOR UPDATE, FOR SHARE, SKIP LOCKED,
458
+ LOCK IN SHARE MODE) is refused.
459
+ - Database writes through a SQL Server synonym or view, or a MySQL view,
460
+ are refused, since they may reach a table in another database.
461
+ - On MySQL, INSERT ... SET is read with its columns, as INSERT ... VALUES
462
+ is.
463
+ - A token in an API response that no step extracts (an access_token or
464
+ refresh_token field, a Location carrying #access_token=) no longer
465
+ reaches the transcript, reports, traces, error lines or the recording
466
+ agent in clear.
467
+ - A polled request (until) keeps polling after a lost connection (a
468
+ reset) or an attempt that timed out, and fails with that error only once
469
+ the poll's time is up. A stopped run, a DNS failure or a TLS failure
470
+ ends the poll at once.
471
+ - The recording agent can no longer record a response-time limit (maxMs)
472
+ or a poll limit (until.timeoutMs) other than the one the scenario states
473
+ in figures: the bound must be a duration the step or the checks right
474
+ after it name, in any unit.
475
+ - A Then whose only check is the 2xx class, or the very status its
476
+ request already required, is refused as a check that could never fail:
477
+ the agent records a field or schema check instead.
478
+ - Two requests at once in a new API session no longer open two clients,
479
+ one of which was never closed.
480
+ - In a parallel run with the agent, a scenario whose replay failed and
481
+ then passed when run again for the agent is red, as with one worker: it
482
+ was reported green, and a failure that comes and goes passed with
483
+ nothing in the report.
484
+ - `saffron run --plan` refuses an invalid `--workers` or
485
+ `--agent-workers`, as a run does.
486
+ - A failed check on a field named as a credential (a token, a session id)
487
+ says `[credential, not kept]` instead of its value, in the error, the
488
+ report and a poll's last reason.
489
+ - Hiding a credential field in a response keeps the rest of the body as
490
+ the API wrote it: large ids, decimals like 1.10 and spacing are not
491
+ rewritten.
492
+ - Exported response checks wait for the page's requests to settle and fail
493
+ when the same request is answered again with a status the check rejects,
494
+ as replay does.
495
+ - Exported "URL does not contain", count and differs checks hold only once
496
+ the page settles, as replay's do: a navigation on its way, a list still
497
+ growing or a value that changes for a moment no longer passes them.
498
+ - Exported captures and differs checks read the displayed copy of an
499
+ element after a hidden one with the same text, as replay does.
500
+ - Exported value checks require a shown input that holds the value in the
501
+ same look.
502
+ - Exported "not shown" checks on a literal name pattern also find the name
503
+ with invisible format characters in it, as replay does.
504
+ - The exported helper file writes the project root with forward slashes on
505
+ Windows, so it reads on macOS and Linux too.
506
+ - A heal of a step that failed partway through is told which of the step's
507
+ actions replay already performed, and records only the rest; a healed
508
+ recording no longer repeats them (an Add click run twice before
509
+ Proceed).
510
+ - Behavior change: a heal that changes nothing in the recording no longer
511
+ files a proposal when its proof passes. Such a run used to be a yellow
512
+ proposal and is now red, saying the recording failed on replay and then
513
+ passed again (timing or the app's state), as it already was with
514
+ --no-verify.
515
+ - A proposal not filed on this machine no longer turns a failed committed
516
+ recording yellow when it checks the Then steps differently from the
517
+ committed recording: the run is red and names the pending file.
518
+ - A check in a browser session that has no page yet (As "Bob") is checked
519
+ on that session's own page, as replay does, never on the default
520
+ session's page.
521
+ - A seeded step that an agent segment went past and recorded nothing for
522
+ keeps its seeded recording instead of an empty one.
523
+ - "Checks proven sensitive" is claimed only when every Then was probed; a
524
+ Then the sensitivity check leaves out (one that clicks, for example)
525
+ leaves it unclaimed.
526
+ - A row value checked through XPath `normalize-space(...)`, `string(...)`
527
+ or `./text()`, or through an unquoted CSS attribute value
528
+ (`[data-k*=x-<id>]`), is kept whole, so row 1 no longer passes or clicks
529
+ on "Order 12". A row value inside another XPath function, such as
530
+ `translate` or `concat`, is refused instead of matched loosely.
531
+ - A "not shown" check whose selector holds the text itself
532
+ (`:has-text(...)`, `text=...`, XPath `contains(., ...)`) fails when the
533
+ page shows that text with invisible characters around the row value,
534
+ such as the bidi isolates Fluent and ICU add.
535
+ - An invisible character (soft hyphen, zero-width space, bidi isolate) no
536
+ longer counts as a word boundary, so "saved" does not pass on "unsaved"
537
+ written with a soft hyphen, and row 1 does not pass on "Order 12" with a
538
+ zero-width space inside.
539
+ - A recorded response, attribute, style or JSON field pattern that
540
+ backtracks fails the step at once with a message saying so, instead of
541
+ freezing the run.
542
+ - A "not shown" check treats straight and typographic quotes alike, so
543
+ "Can't save" fails while the page shows "Can’t save".
544
+ - Exported Playwright tests fail a recorded pattern that backtracks (a
545
+ URL, a response body or JSON field, an attribute, a style) saying so, as
546
+ replay does, no longer take an invisible format character as a word
547
+ boundary, and read straight and typographic quotes alike in "not shown"
548
+ checks.
549
+ - A credential in an API base URL's query (`api.baseURL` or `--api-url`
550
+ with `?api_key=`) is masked in request errors, the transcript and what
551
+ the recording agent reads, as the URL's password already was; one in the
552
+ browser `baseURL` query (`?token=`) is masked in traces too.
553
+ - A failed header check never quotes a credential header's value
554
+ (Set-Cookie) and masks a token in a Location it quotes.
555
+ - A row value inside an XPath `or` (`contains(., "Order <id>") or
556
+ contains(., "Order #<id>")`) is refused as one replay cannot tie to its
557
+ row, as inside `translate` or `concat`: a "not shown" check passed on
558
+ the other branch.
559
+ - A "not shown" check through `:text(...)`, `:text-is(...)`,
560
+ `:text-matches(...)`, XPath `normalize-space(.)="..."` or
561
+ `text()="..."`, or an exact `internal:text="..."s` also finds the text
562
+ with invisible characters around the row value.
563
+ - A numeric one-time code or PIN in an API response (`"otp": 123456`) is
564
+ hidden in the transcript and error lines, as a text one is.
565
+ - A recorded pattern bounds a row value before `.*`, `[^,]*` or `.{0,9}`
566
+ and more text: `/Order <id>.*Shipped/` for row 1 passed on "Order 12
567
+ Shipped". An optional or loose point after the value (`<id>\.?`,
568
+ `<id>[,;]`) no longer lets "1.5" or "1,000" pass for 1.
569
+ - "Not shown" finds text in markup with no space between its elements
570
+ (React, Vue, minified HTML): "Order 1" passed as gone while the row read
571
+ "Order 1Pending", and "Bread" while it read "BreadIn stock". A "not
572
+ shown" by role and name on a role whose name never comes from its text (a
573
+ list item, a status, an alert, a paragraph) reads its text: it passed
574
+ with the text still shown.
575
+ - An attribute check on `checked`, `selected` or `value` reads the control
576
+ as it is now, in replay and in exported tests: a box the app unticked
577
+ passed as ticked.
578
+ - Patterns, captures and amounts never read across a block's edge, as text
579
+ checks and exported tests do: "Qty 1" over "0 left" no longer reads "Qty
580
+ 10", nor "$1" over "5 nights" the amount 15. A price split into flex items
581
+ ("$" "12" ".50") still reads "$12.50".
582
+ - A row value written unquoted inside an XPath function
583
+ (`contains(@data-order, <id>)`) is refused: row 1 found order 12.
584
+ - A row's number is never written as `<column>` inside a longer number
585
+ ("9.5" in "€9.50", "104" in "order-000104"): another row's "not shown"
586
+ looked for "€12.250" and passed. A zero-padded id in a locator now binds
587
+ the recording to its row, so other rows are red until the step names the
588
+ id as the page writes it.
589
+ - A two-column table with a repeated first cell is judged whole, so an
590
+ edit to an earlier row re-records and seeds no other scenario.
591
+ - A long number an Outline row's page sets in its own way ("+4712345678"
592
+ shown "+47 123 45 678", an IBAN in blocks) binds the recording to its
593
+ row, with guidance, where another row passed on it.
594
+ - The recording agent cannot check a value against one captured in the same
595
+ Then, nor a response by a pattern every URL matches, nor bind a database
596
+ check to part of a row's value ("ann" of ann@example.test). A watch armed
597
+ in a step replay ran before the session can be checked.
598
+ - A recording the post-process refuses is told to the agent and ends as an
599
+ ordinary red, its cost counted, not as the provider failing.
600
+ - An API request whose path segment comes out blank (`/orders/{captured:id}`
601
+ with an empty id) is refused, in replay and in exported tests: it went to
602
+ the collection URL and passed. A credential a response echoes under a
603
+ plain name is kept from other origins.
604
+ - Database checks refuse MySQL `||` (run as OR), a WITH that selects a bind
605
+ itself, and on PostgreSQL a backslash before a quote; "at most 0" on a
606
+ count is an absence check, as "equals 0" is.
607
+ - A kept trace no longer holds the saved login (`storageState`), basic-auth
608
+ credentials or extra headers, and every cookie in its network log is
609
+ redacted whatever its name. A `{refresh, access}` token pair's access
610
+ token is a credential: hidden from the agent, the transcript and check
611
+ failures, and kept from other origins.
612
+ - A pinned contract applies `required` on a `readOnly` property to
613
+ responses only and on a `writeOnly` one to requests only, so one shared
614
+ model no longer refuses every request body. A response the contract
615
+ gives as `*/*` is checked against its schema.
616
+ - `saffron status`, `saffron coverage` and the editors keep the last run's
617
+ result of each scenario when the report came from a CI machine on another
618
+ OS.
619
+ - `--rerecord` puts back a recording it removed when the scenario was not
620
+ recorded again (the app down, a capability off, the budget spent,
621
+ Ctrl-C). `run --plan --rerecord` replays a fixture, as the run does.
622
+ - `saffron report --merge --strict` exits 1 on a pending review, as shards
623
+ run with `--strict` do.
624
+ - A parallel run's kept red no longer names evidence its second try
625
+ deleted.
626
+ - A feature edit that would drop an Outline `<column>` is skipped, and a
627
+ file with mixed line endings keeps every other line's ending.
628
+ - A short config key of the team's own (`app`, `id`) is ignored, not
629
+ refused as a misspelling; a short key's letters swapped (`evn`) still is.
630
+ - `differs from` compares `$5` and `5 USD` as one currency.
631
+
632
+ Editors (VS Code extension 0.8.2, JetBrains plugin 0.8.2)
633
+
634
+ - Database steps: a step recorded with a database check or write is not
635
+ offered as a zero-token wording to reuse (completion, the step hover and
636
+ StepSet expansion say it replays at zero tokens only in its own
637
+ scenario), and a step is marked as one only when every recording of it
638
+ reads or writes the database.
639
+ - A proposal with the runner's new "database check repaired after a schema
640
+ change:" or "database write added by the heal:" note is held from Accept
641
+ All, as `saffron accept --all` skips it.
642
+ - The last run's line, a CI run's totals and its rows count a red the app
643
+ did not cause (a stale recording without the agent, a missing
644
+ environment, the agent or the budget) as not run, as the runner's status
645
+ does.
646
+ - Open trace finds a CI scenario whose feature path the runner left out,
647
+ by its name.
648
+ - Eject's This file runs in the open file's own Saffron project, with
649
+ forward slashes on Windows, and is not offered for a file outside every
650
+ project.
651
+ - VS Code: a status or diff that times out is stopped with its child
652
+ processes, a cut diff says so, and `{data:...}` hints read the
653
+ environment as the runner does. JetBrains: the Dashboard reloads only on
654
+ new runner output, and proposals and lenses no longer read the disk on
655
+ the UI thread.
656
+
8
657
  ## 1.3.5: Scenarios kept from running at once
9
658
 
10
659
  Released 2026-11-08.
@@ -1606,7 +2255,7 @@ Runner
1606
2255
  newest marked. A duration over a minute reads as minutes and seconds,
1607
2256
  and scenarios sharing a cause get a button for the command that deals
1608
2257
  with them all, the command itself still on the page.
1609
- - **A compatibility page per release.** [Compatibility](COMPATIBILITY.md)
2258
+ - **A compatibility page per release.** [Compatibility](docs/COMPATIBILITY.md)
1610
2259
  lists the Node.js, operating systems, Playwright and browsers, each
1611
2260
  agent's SDK (the range accepted and the version tested) and the recording
1612
2261
  versions this release reads and writes, generated from the release's own
@@ -1618,7 +2267,7 @@ Runner
1618
2267
  greens); `--record` also records the suite afresh with the agent and
1619
2268
  measures the share verified on the first attempt, the share held for
1620
2269
  review, and the time and cost. The results are published in the docs
1621
- ([benchmark](BENCHMARK.md)): today 66 of 66 replays green and no false
2270
+ ([benchmark](docs/BENCHMARK.md)): today 66 of 66 replays green and no false
1622
2271
  green on 8 broken apps.
1623
2272
  - **A coverage map per pull request.** `saffron coverage` maps each
1624
2273
  requirement ID (a tag matching `requirementTag`, such as `@REQ-12`) to