saffron-ai 1.3.4 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +745 -2
- package/README.md +18 -4
- package/dist-pkg/cli.js +450 -425
- package/package.json +26 -2
- package/skills/saffron/SKILL.md +40 -7
- package/skills/saffron/references/api.md +8 -1
- package/skills/saffron/references/config.md +5 -4
- package/skills/saffron/references/database.md +164 -0
- package/skills/saffron/references/syntax.md +32 -2
- package/templates/playwright/saffron-helpers.ts +346 -69
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,749 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.4.0: Database steps
|
|
9
|
+
|
|
10
|
+
Released 2026-11-10.
|
|
11
|
+
|
|
12
|
+
Runner
|
|
13
|
+
|
|
14
|
+
- Database steps (beta): a step like `Then the customer should
|
|
15
|
+
be saved in the database` works with no query written by hand. While
|
|
16
|
+
recording, the agent reads the schema and records one parameterized
|
|
17
|
+
`SELECT`, its values as binds; every later run replays it at zero
|
|
18
|
+
tokens. A check must be bound to a value the scenario wrote; a value that
|
|
19
|
+
is new on each run (`{unique:...}`, an id an earlier step kept) keeps it
|
|
20
|
+
from passing on rows an earlier run left. With `"writes": true` and the
|
|
21
|
+
database's host or name in `allow`, a `Given` or `When` may seed data with
|
|
22
|
+
one `INSERT` or `UPDATE ... WHERE`. When the scenario ends, green or
|
|
23
|
+
red, `db.cleanup` undoes them: `"all"` (default) deletes the rows it
|
|
24
|
+
inserted and puts the rows it updated back as they were, `"inserts"`
|
|
25
|
+
deletes inserts only, `"none"` leaves everything. `DELETE`, `TRUNCATE`,
|
|
26
|
+
DDL and several statements are refused by a SQL parser. Set `db` in
|
|
27
|
+
`saffron.config.json` and `npm install node-sql-parser` with the
|
|
28
|
+
engine's driver: neither comes with saffron-ai, so a project without
|
|
29
|
+
database steps does not carry the parser's 88 MB. Recordings with
|
|
30
|
+
database steps are cache version 20.
|
|
31
|
+
- Database steps on PostgreSQL, MySQL and SQL Server, named by db.url's
|
|
32
|
+
scheme (`postgres://`, `mysql://`, `mssql://`), with the driver `pg`,
|
|
33
|
+
`mysql2` or `mssql`. Each engine takes the same steps, rules and `$1` binds; MySQL and SQL Server writes
|
|
34
|
+
are read back by their primary key, one row per `INSERT` when the key
|
|
35
|
+
is auto-numbered. A recording made on one engine is refused on another.
|
|
36
|
+
A write to more than one table (MySQL's `UPDATE a JOIN b`) is refused,
|
|
37
|
+
and a write whose rows cannot be read back by their key is rolled back,
|
|
38
|
+
so cleanup never misses a row or touches one Saffron did not write. A
|
|
39
|
+
SQL Server check that meets a row the app is still writing reads again
|
|
40
|
+
instead of failing. A table hint (`WITH (NOLOCK)`, `READPAST`) is refused,
|
|
41
|
+
so a check never reads rows the app did not commit or skips rows it
|
|
42
|
+
holds. A write goes only to the database `db.url` names (MySQL's
|
|
43
|
+
`otherdb.t` and SQL Server's `db.schema.t` are refused), an `UPDATE` with
|
|
44
|
+
`ORDER BY` or `LIMIT` is refused, and a SQL Server `UPDATE` that changes a
|
|
45
|
+
time finer than milliseconds, or with an offset, is refused while cleanup
|
|
46
|
+
would put it back. MySQL's `INSERT IGNORE` is refused, as it skips rows
|
|
47
|
+
without saying so, and a check whose rows the app holds locked for the
|
|
48
|
+
whole `db.timeout` fails saying so.
|
|
49
|
+
- A database check reads again within `db.timeout` (default 3000 ms): a
|
|
50
|
+
presence check passes as soon as its rows are there, and a "not in the
|
|
51
|
+
database" check waits for a row a worker deletes late, then must hold
|
|
52
|
+
to the end of the window, so a row a worker writes late fails it.
|
|
53
|
+
- On PostgreSQL and MySQL every statement a database step runs goes to
|
|
54
|
+
the database as exactly one statement, so a query the SQL check reads
|
|
55
|
+
as one SELECT can never run a second statement (a quote read differently
|
|
56
|
+
by the two could hide a COMMIT and a DELETE in a look at the data). SQL
|
|
57
|
+
Server runs a batch, so there the parser's one-statement rule and the
|
|
58
|
+
refused backslash before a quote hold it.
|
|
59
|
+
- Cleanup puts a json or jsonb column back as JSON, an array or a string
|
|
60
|
+
value included. Placeholders numbered with a gap ($1 and $3, no $2) are
|
|
61
|
+
refused while recording.
|
|
62
|
+
- Only a scenario whose recording has database steps, or whose steps name
|
|
63
|
+
the database before it is recorded, needs the database to answer before
|
|
64
|
+
recording starts.
|
|
65
|
+
- A database check that the database refused (a permission, a timeout)
|
|
66
|
+
says so, apart from one that found other rows; neither starts an agent.
|
|
67
|
+
- A database check whose stored query fails because the schema changed
|
|
68
|
+
is repaired by the agent, the one exception to a `Then` never being
|
|
69
|
+
healed: only its table, columns and joins may change, its binds and
|
|
70
|
+
expected rows are enforced unchanged, it must return as many rows as
|
|
71
|
+
the recording did (a join that fans out is refused), and the repair must
|
|
72
|
+
pass a proof replay. The step is reported healed and a proposal names the old and
|
|
73
|
+
new SQL. A query that returns the wrong rows stays red, with no agent.
|
|
74
|
+
- The run report shows the rows each database check found under its
|
|
75
|
+
step, masked, the first five.
|
|
76
|
+
- A recorded check must tie its rows to the scenario's value: a column
|
|
77
|
+
compared with a bind by `=`, `IN` or `LIKE`, on each side of an `OR`,
|
|
78
|
+
and a subquery on the tying side ties its own rows the same way.
|
|
79
|
+
`UNION`, `INTERSECT` and `EXCEPT`, `email <> $1`, `NOT (...)`, a
|
|
80
|
+
comparison inside `COALESCE` or `CASE`, a range alone
|
|
81
|
+
(`created_at > $1`) and a bind in `LIMIT` or `OFFSET` are refused, as
|
|
82
|
+
each could hold on rows the scenario never wrote. The same holds for an
|
|
83
|
+
`UPDATE`'s `WHERE`. A JSON field can tie a check
|
|
84
|
+
(`data->>'orderRef' = $1`, `data @> CAST($1 AS jsonb)`). A null bind,
|
|
85
|
+
or an empty one in a "not in the database" check, is refused while
|
|
86
|
+
recording and at replay: `= NULL` matches no row.
|
|
87
|
+
- A repaired database check, and a heal that adds or changes a database
|
|
88
|
+
write, are held for review: `accept --all` leaves them out (`DATABASE
|
|
89
|
+
CHECK REPAIRED`, `DATABASE WRITE ADDED`).
|
|
90
|
+
- A database that is down, restarts mid-step or does not answer within 10
|
|
91
|
+
seconds is the environment: no agent session starts, and the message
|
|
92
|
+
says the database did not answer, not the app. Before recording, an
|
|
93
|
+
unset `db.url` variable, a connection string that is not a URL, a wrong
|
|
94
|
+
password, a missing database and a missing driver (`pg`, `mysql2` or
|
|
95
|
+
`mssql`) or `node-sql-parser` are said at once, without starting an
|
|
96
|
+
agent.
|
|
97
|
+
- `saffron status --json` and the `project_status` MCP tool say whether
|
|
98
|
+
database steps are configured (`database`: the engine, whether writes
|
|
99
|
+
are on, the cleanup mode), never the connection string; they and the
|
|
100
|
+
editors' config views mask a password in a `db.url` written in keyword
|
|
101
|
+
form (`host=... password=...`) or in its query (`?password=`). The
|
|
102
|
+
bundled skill, `saffron init`'s AGENTS.md block and the guides cover
|
|
103
|
+
database steps and the one Then whose query may be repaired.
|
|
104
|
+
- `run --plan` and the MCP `run_plan` tool list a recorded database write
|
|
105
|
+
as a side effect when `db.cleanup` keeps it (`"inserts"` keeps an
|
|
106
|
+
`UPDATE`, `"none"` every write).
|
|
107
|
+
- An `@api` scenario whose recording sends no request (database steps
|
|
108
|
+
alone) replays without the API answering.
|
|
109
|
+
- The language server (JetBrains completion and hover) says a step whose
|
|
110
|
+
recordings all read or write the database replays at zero tokens only in
|
|
111
|
+
the scenario that recorded it, per `As "..."` session, as VS Code does,
|
|
112
|
+
and its similar-wording hint no longer suggests such a step.
|
|
113
|
+
- Found in the release review, before release:
|
|
114
|
+
- A write inside a `WITH` passed when both statements were `UPDATE`s,
|
|
115
|
+
and its table was never put back. Any `WITH` in a write is refused,
|
|
116
|
+
and so is a `WHERE` that reads no column (`WHERE true`).
|
|
117
|
+
- A check could echo its bind back (`SELECT $1 AS email`), or hold on
|
|
118
|
+
any row (`OR true`, `OR email IS NOT NULL`, `(email = $1) IS NOT NULL`,
|
|
119
|
+
`$1 = $1`), and a schema repair could add
|
|
120
|
+
`LIMIT 1`, `DISTINCT` or `max()` to hide a join that fans out. Both
|
|
121
|
+
are refused, and so are values written in as `E'...'`, `$$...$$` or
|
|
122
|
+
`DATE '...'`.
|
|
123
|
+
- Under `"assertionPolicy": "adaptable-mid"`, a heal rewrote a database
|
|
124
|
+
check in the middle of a scenario, its binds and expected rows too.
|
|
125
|
+
A database check is never healed, under every policy.
|
|
126
|
+
- A database check recorded by another scenario was reused with that
|
|
127
|
+
scenario's value, so it could pass on rows an earlier run left. A
|
|
128
|
+
database step is seeded only from the scenario's own recording.
|
|
129
|
+
- A `db.url` written as `host=... password=...` was shown whole in the
|
|
130
|
+
error, password included. The message no longer shows the value, and
|
|
131
|
+
the rows an error shows are masked before they are cut.
|
|
132
|
+
- A database connection dropped mid-step crashed the run, a host that
|
|
133
|
+
does not answer now fails within 10 seconds, and a step still running
|
|
134
|
+
after Ctrl-C opens no new connection.
|
|
135
|
+
- Ctrl-C during a database write left its row behind. Closing now
|
|
136
|
+
waits for the write and undoes it.
|
|
137
|
+
- A check on an aggregate's one row passed whatever the table held:
|
|
138
|
+
`count(*)` with no `GROUP BY`, an aggregate the parser reads as a
|
|
139
|
+
plain function (`json_agg`, `JSON_ARRAYAGG`, `bool_or`), `GROUP BY
|
|
140
|
+
ROLLUP`, and a query whose only `FROM` or `WITH` source is an
|
|
141
|
+
aggregate. A count, exists or item check on that row is refused while
|
|
142
|
+
recording and at replay; only a value checked on it is taken.
|
|
143
|
+
- A count above 0, or a check that there is no second row, could not see
|
|
144
|
+
a second row under `LIMIT`, `TOP`, `DISTINCT` or a `GROUP BY` with no
|
|
145
|
+
aggregate, and `OFFSET` (SQL Server's `OFFSET ... FETCH` too) hid rows
|
|
146
|
+
from a count and let a "no rows" check pass: each is refused. On MySQL
|
|
147
|
+
and SQL Server a check never saw `DISTINCT` (the parser writes it
|
|
148
|
+
differently there), so such checks passed on a duplicate row and a
|
|
149
|
+
repair could add it; it is now read on every engine.
|
|
150
|
+
- A check that expects no rows, or that a field must not be there, is no
|
|
151
|
+
longer repaired: a repaired query finding none proved nothing, and a
|
|
152
|
+
repair could stop selecting that field.
|
|
153
|
+
- A field that must not be in a row (`/0/data/discount`) passed when the
|
|
154
|
+
app saved no row at all; it now needs the row to be there.
|
|
155
|
+
- A captured id of one or two characters was recorded as itself in a
|
|
156
|
+
database bind, so replay checked the recording run's row; it must be
|
|
157
|
+
written as `{captured:name}`, and one written as itself beside another
|
|
158
|
+
bind is refused.
|
|
159
|
+
- On MySQL, `--` with no space after it let a look at the data run what
|
|
160
|
+
Saffron read as a comment; it is refused. On SQL Server, the agent's
|
|
161
|
+
`UPDATE` was read as PostgreSQL when checking that its `WHERE` names
|
|
162
|
+
the scenario's rows, and a write to a table whose key the driver reads
|
|
163
|
+
inexactly (a decimal past 15 digits) is refused while cleanup is on.
|
|
164
|
+
- A page fingerprint left by a browser tool that failed went onto the
|
|
165
|
+
next database or HTTP action, and that recording was refused on every
|
|
166
|
+
load after.
|
|
167
|
+
- A MySQL write could read back a row that was there before (`7`
|
|
168
|
+
finding `'07'`) for one its key missed, and cleanup deleted it: each
|
|
169
|
+
key now reads back exactly its own row, or the write is rolled back.
|
|
170
|
+
A JSON column is put back as the text it held, never read through a
|
|
171
|
+
JS number.
|
|
172
|
+
- A secret a driver quotes cut short (MySQL's duplicate entry at 64
|
|
173
|
+
characters, SQL Server's truncated value), one holding a parenthesis
|
|
174
|
+
or cut to 4 characters too, is masked, and a secret in a row no longer
|
|
175
|
+
breaks the rows the report shows. A lock held for the whole window
|
|
176
|
+
says it timed out, not that it found other rows.
|
|
177
|
+
- A database check was seeded from the scenario's own recording after
|
|
178
|
+
an earlier step changed the value it binds; it is recorded again. A
|
|
179
|
+
stale recording whose edited steps name the database now asks the
|
|
180
|
+
database to answer before recording.
|
|
181
|
+
- A SQL Server `BIGINT` identity past 2^53 is read exactly, so cleanup
|
|
182
|
+
deletes the scenario's row, never the app's.
|
|
183
|
+
- A check or an agent `UPDATE` with an `OR` whose other side is a fixed
|
|
184
|
+
value (`email = $1 OR role = 'admin'`) passed on rows earlier runs
|
|
185
|
+
left: each side of an `OR`, and every value of an `IN`, must hold a
|
|
186
|
+
value the scenario wrote. A `LIKE` value of only `%` or `_` (from an
|
|
187
|
+
Examples cell too) is refused unless another condition ties the rows,
|
|
188
|
+
and so are `LIMIT 0`, `TOP 0` and `FETCH ... 0`.
|
|
189
|
+
- A quoted, qualified column (`` `o`.`total` ``, `"o"."total"`) was
|
|
190
|
+
refused as a value written into the SQL; it is read as a name.
|
|
191
|
+
- Stability proofs and an Outline's row proofs failed on rows the first
|
|
192
|
+
proof seeded: its database writes are now undone before they run.
|
|
193
|
+
- The MCP `search_steps` tool showed the agent a database step's
|
|
194
|
+
session-prefixed text (`As "name" ...`), which would move that step
|
|
195
|
+
into the named session; it no longer does.
|
|
196
|
+
- An Outline check that holds a row's value as the page sets it in type
|
|
197
|
+
(a curly apostrophe or quotes, spaces collapsed or no-break, an
|
|
198
|
+
ellipsis) let another row pass on it; that row now records again or is
|
|
199
|
+
refused.
|
|
200
|
+
- A key under `api` or `api.services.<name>` that Saffron does not read
|
|
201
|
+
(`"transcripts"`, `"contracts"`) is refused with the keys it takes,
|
|
202
|
+
instead of leaving that setting at its default.
|
|
203
|
+
- A check that something is shown (text, visible, a field's value) no
|
|
204
|
+
longer passes on content inside a frame the page hides (`visibility:
|
|
205
|
+
hidden`, zero-sized, placed off the page), or inside a frame in one.
|
|
206
|
+
- A base URL's password, which Chromium keeps in the page's URL, is
|
|
207
|
+
masked in what the recording agent reads: snapshots, tool errors,
|
|
208
|
+
Saffron tool replies and the heal prompt's live page and failure.
|
|
209
|
+
Navigating to the masked URL still signs in.
|
|
210
|
+
- A remote browser's endpoint (`transport.browser`) no longer prints its
|
|
211
|
+
password or `?token=` key when the connection fails.
|
|
212
|
+
- `saffron status --json`, the `project_status` and `api_contract` MCP
|
|
213
|
+
tools mask a password in a base URL and a key in a URL's query or
|
|
214
|
+
fragment (`?token=`, `?api_key=`, `?key=`, `#access_token=`); `api_contract` showed the password
|
|
215
|
+
to the agent. The MCP `verify` tool masks them in a failed run's last
|
|
216
|
+
lines, and `saffron api pin` in what it prints.
|
|
217
|
+
- `saffron api pin` says why a URL could not be fetched (connection
|
|
218
|
+
refused, host not found) instead of only "fetch failed".
|
|
219
|
+
- `saffron run --json` without `--plan` exits 2 instead of starting a run
|
|
220
|
+
that prints no JSON.
|
|
221
|
+
- `saffron eject --out <dir>` warns when the project's Playwright config
|
|
222
|
+
looks for tests in another folder, so `npx playwright test` would not
|
|
223
|
+
run the new specs.
|
|
224
|
+
- `saffron init` registers the MCP server as `npx -y -p saffron-ai saffron
|
|
225
|
+
mcp`: a bare `npx saffron` fetched npm's unrelated `saffron` package in
|
|
226
|
+
a project without saffron-ai installed.
|
|
227
|
+
- A `saffron.config.json` that is not valid JSON is named in the error.
|
|
228
|
+
- `saffron status --json` and the `project_status` MCP tool name, in each
|
|
229
|
+
`lastRun.attention[]` entry, why a red was not run (`blocked`:
|
|
230
|
+
`recorder`, `environment`, `provider` or `budget`), so editors list it
|
|
231
|
+
as not run instead of failed.
|
|
232
|
+
- `--project-root` naming a folder that does not exist stops with exit 2,
|
|
233
|
+
instead of reading it as an empty project, and `saffron run` names the
|
|
234
|
+
configured features folder when it does not exist.
|
|
235
|
+
- A run notes when the selected data environment (`--env`, `SAFFRON_ENV`
|
|
236
|
+
or `env`) matches no data file, so a misspelled name no longer uses the
|
|
237
|
+
base data in silence.
|
|
238
|
+
- `Examples: {data:...}` refuses a CSV file that names a column twice, as
|
|
239
|
+
an Examples table does; one of them was read for both. A data JSON file
|
|
240
|
+
saved with a byte order mark is read.
|
|
241
|
+
- The recording agent can no longer send an `{env:...}` value to another
|
|
242
|
+
site by writing the site as a reference (`{captured:link}?p={env:...}`,
|
|
243
|
+
a row's `<column>`, a `{data:...}` value): the site is checked as the
|
|
244
|
+
browser will open it.
|
|
245
|
+
- A capture, and a check that a value changed, wait while the element
|
|
246
|
+
they were recorded on is still empty, instead of reading the same test
|
|
247
|
+
id in an iframe or a fallback that happens to show text: "the price
|
|
248
|
+
changed" passed on an embedded widget's copy.
|
|
249
|
+
- When a tab closes (a step closes it, or it closes itself, as a sign-in
|
|
250
|
+
popup does), replay and an exported Playwright test go where the
|
|
251
|
+
recording's browser went: they stay where they were when another tab
|
|
252
|
+
closed, and take the tab that slid into the closed one's place, never
|
|
253
|
+
simply the last tab.
|
|
254
|
+
- `saffron eject` no longer writes a base URL's password, or a key in its
|
|
255
|
+
query or fragment (`?api_key=`, `#access_token=`), into
|
|
256
|
+
`playwright.config.ts` or `saffron-helpers.ts`: it writes `***` and
|
|
257
|
+
warns, naming the variable to set to the full URL (`BASE_URL`,
|
|
258
|
+
`API_URL` or `API_URL_<NAME>`).
|
|
259
|
+
- When a click opens a new tab and the agent keeps working on the page it
|
|
260
|
+
clicked (the browser it records with stays there), the recording now
|
|
261
|
+
switches back to that page, so replay runs those steps there instead of
|
|
262
|
+
on the new tab, where a "not shown" check could pass on the wrong page.
|
|
263
|
+
A tab closed after such a click is closed from the agent's page too, and
|
|
264
|
+
a heal that takes over on a tab replay followed records a switch to the
|
|
265
|
+
tab the agent works on (its browser starts on the first tab). A
|
|
266
|
+
recording made earlier keeps its behavior: re-record a scenario that
|
|
267
|
+
opens a tab and then acts on the page that opened it.
|
|
268
|
+
- Replay follows only a tab a recorded action opened. One that opens
|
|
269
|
+
without such an action (a popup more than 5 seconds after its click, a
|
|
270
|
+
page timer) is not followed, as the agent's browser stayed on its page:
|
|
271
|
+
the checks after it ran on the new tab, where they could pass. Exported
|
|
272
|
+
tests follow tabs the same way.
|
|
273
|
+
- An older recording that uses tabs without tab marks still follows any
|
|
274
|
+
new tab, also one that opens later than 150 ms after a click: the check
|
|
275
|
+
after it moves to the new tab, and an action waits for the tab or its
|
|
276
|
+
element, a `{unique:...}` target included. A click that opens no tab is
|
|
277
|
+
not slowed. A "not shown" check there must hold on the page it started
|
|
278
|
+
on, then waits for the tab and must hold on it too, so neither page
|
|
279
|
+
passes it for the other (record the scenario again to add tab marks and
|
|
280
|
+
drop the wait).
|
|
281
|
+
- Exported tests wait for a tab a background click opened, and take a tab
|
|
282
|
+
that already opened before a switch or close, as replay does.
|
|
283
|
+
- Reports and run history mask a key in the base URL's query or fragment
|
|
284
|
+
(`?api_key=`, `#access_token=`), as they already masked its password.
|
|
285
|
+
- A signed storage URL's signature (an Azure SAS `?sig=`) is masked
|
|
286
|
+
wherever a key in a URL's query is.
|
|
287
|
+
- `saffron eject` warns when the exported tests send API requests while
|
|
288
|
+
`transport.api` or `transport.module` is set: they go without its
|
|
289
|
+
headers, proxy or certificate setting.
|
|
290
|
+
- A feature path a Windows machine recorded (`features\login.feature`)
|
|
291
|
+
is found on macOS and Linux: `saffron prune` deleted such a recording of
|
|
292
|
+
a live feature outside the features folder, `saffron import` refused a
|
|
293
|
+
Windows CI run's proposals as "not in this checkout", `saffron trace`
|
|
294
|
+
called them stale, `accept` could not apply their feature edits, and
|
|
295
|
+
viewing that run hid "Open scenario". `saffron coverage` writes feature
|
|
296
|
+
paths with forward slashes, and `--base` reads a map committed on
|
|
297
|
+
Windows as the same files instead of every scenario removed.
|
|
298
|
+
- Feature paths written on Windows also match elsewhere in the run: a
|
|
299
|
+
sharded run split across Windows and Linux machines gives every
|
|
300
|
+
scenario to exactly one shard (some ran twice and some never), `saffron
|
|
301
|
+
diff <feature file>` finds a proposal filed on Windows, `run --plan`
|
|
302
|
+
treats such a pending proposal as the run does instead of planning a
|
|
303
|
+
recording, and `status --json` names an Examples row's scenario
|
|
304
|
+
(`baseScenario`) from a report written on Windows. `saffron init` keeps
|
|
305
|
+
the Windows line endings of an AGENTS.md or CLAUDE.md that uses them.
|
|
306
|
+
- A dropdown option picked in a form is replayed by its label when
|
|
307
|
+
another option's value is the same text (that earlier option was chosen
|
|
308
|
+
instead). Exported tests do the same.
|
|
309
|
+
- Recording with many agent workers no longer prints a
|
|
310
|
+
MaxListenersExceededWarning: the sessions share one exit listener.
|
|
311
|
+
- A fixture whose pending proposal is deleted or unreadable mid-run (with
|
|
312
|
+
proofs off) makes its scenarios red, naming why, instead of ending the
|
|
313
|
+
run without a report.
|
|
314
|
+
- `saffron accept` writes a feature edit atomically, keeping the file's
|
|
315
|
+
line endings, permissions and symlink, so a crash mid-write no longer
|
|
316
|
+
leaves it cut short.
|
|
317
|
+
- One StepSet file that does not parse, or defines a set twice, no longer
|
|
318
|
+
stops `saffron steps` and the MCP `search_steps` and `list_step_sets`
|
|
319
|
+
tools: the other files' steps are offered and that file is named.
|
|
320
|
+
- The language server refreshes `{data:...}` hints when files in a
|
|
321
|
+
configured `dataDir` change, not only in `data/`.
|
|
322
|
+
- Recording keeps right, middle and Ctrl or Shift clicks: Playwright MCP
|
|
323
|
+
writes their options over several lines, and they were left out of the
|
|
324
|
+
recording.
|
|
325
|
+
- A `{unique:name:digits}` value sent as a number (in a request body or a
|
|
326
|
+
database check bind) is recorded as its token: it was kept as the
|
|
327
|
+
recording run's number, so later runs checked that run's row.
|
|
328
|
+
- An abbreviated amount (`€1.5M`, `$1.2K`), or one written in words or
|
|
329
|
+
spaced off ("3 bn", "€1.5 million", "1,5 Mio. €", "5 lakh"), is refused
|
|
330
|
+
instead of read as its digits, and Indian lakh grouping
|
|
331
|
+
(`₹1,23,456.00`) reads as 123456, in replay and exported tests.
|
|
332
|
+
- Copies of a request sent at once (`concurrent`) are never sent again by
|
|
333
|
+
`retries`: a retry could pass a race the first copies lost.
|
|
334
|
+
- Response bodies are read in the charset their Content-Type names, for
|
|
335
|
+
API requests, the browser's response checks and exported tests, and a
|
|
336
|
+
leading byte order mark is ignored: a Latin-1 body read garbled, so a
|
|
337
|
+
`differs` check could pass on an unchanged value.
|
|
338
|
+
- A recording the agent's provider failed (a sign-in that expired, the
|
|
339
|
+
network, a usage limit) is counted as blocked by the agent's provider,
|
|
340
|
+
not as failed in the app.
|
|
341
|
+
- Stopping a run (Ctrl-C, a cancelled CI job) also stops a running Codex
|
|
342
|
+
or Antigravity session, which kept running and spending tokens.
|
|
343
|
+
- `maxSessionMinutes` also stops a session that makes no tool call (a
|
|
344
|
+
vendor waiting on a sign-in, a stalled model call), two minutes past its
|
|
345
|
+
budget, and says why. Such a recording counts as failed, not as blocked
|
|
346
|
+
by the provider, as the app may be what kept it waiting.
|
|
347
|
+
- `saffron import` refuses bundles whose manifests, reports and proposals
|
|
348
|
+
would unzip to more than 1 GB together, before reading any, and masks a
|
|
349
|
+
password or token in a bundle URL wherever it prints it.
|
|
350
|
+
- Recording keeps the checkboxes and radio buttons the agent sets while
|
|
351
|
+
filling a form, a key press of `{`, `(`, `[`, `,`, `'` or `\`, and a
|
|
352
|
+
navigation to an address with an apostrophe; each was dropped without a
|
|
353
|
+
warning, and replay never did it.
|
|
354
|
+
- `saffron trace` refuses a trace file that unzips past its declared size
|
|
355
|
+
or over 1 GB, so a damaged or hostile trace from CI cannot fill memory.
|
|
356
|
+
- A "not shown" check passed while the text was on screen with an
|
|
357
|
+
invisible format character in it: the bidi isolates Fluent and ICU put
|
|
358
|
+
around a value, a soft hyphen, a word joiner. These now count as shown.
|
|
359
|
+
- A step seeded from another with a new quoted value wrote it into a
|
|
360
|
+
pattern unescaped: `matches "Paid (partial)"` passed on "Paid partial".
|
|
361
|
+
- A text check whose expected value holds a captured credential
|
|
362
|
+
(`{captured:...}`) is no longer handed to the page's own scripts, as
|
|
363
|
+
`{env:...}` values already were, so a script on the page or in an iframe
|
|
364
|
+
cannot read it.
|
|
365
|
+
- A closed single dropdown reads as its selected option only: a text,
|
|
366
|
+
pattern, capture or "differs" check no longer passes on another option,
|
|
367
|
+
and a "not shown" check fails while the dropdown shows the option it
|
|
368
|
+
names. A text target ("I see Cancelled") never matches an option the
|
|
369
|
+
dropdown only offers. A list check still reads every option.
|
|
370
|
+
- A "not shown" check by role and name fails when the page shows that name
|
|
371
|
+
in another case ("CANCEL ORDER").
|
|
372
|
+
- Seeding by quoted value keeps every date token form (`{date}`,
|
|
373
|
+
`{date+1}`, `{date+1:D MMM}`) as written in a pattern.
|
|
374
|
+
- A multipart request never sends `saffron.config.json` (in any letter
|
|
375
|
+
case), which may hold a base URL's password.
|
|
376
|
+
- A CI run's results never offer to open a feature path outside the
|
|
377
|
+
project.
|
|
378
|
+
- The language server reads the data environment as a run does: a
|
|
379
|
+
`SAFFRON_ENV` set in the shell, even empty, keeps the `.env` one out.
|
|
380
|
+
- A request the page started before an action (a long-poll opened on
|
|
381
|
+
load) is no longer taken for the action's own when its event arrives a
|
|
382
|
+
moment late, so "not shown", count and response checks no longer wait it
|
|
383
|
+
out: four such checks took 14 seconds instead of 2.
|
|
384
|
+
- A check that a dropdown offers an option reads the option itself
|
|
385
|
+
(`expectText` on `option "Cancelled"`). A 1.3.x check on the dropdown
|
|
386
|
+
that passed on one of its other options is now red: record it again.
|
|
387
|
+
- Under `--no-agent`, a run where the app or the database does not answer
|
|
388
|
+
marks those scenarios blocked by the environment, as a run with the
|
|
389
|
+
agent does, so CI tells an outage from a regression.
|
|
390
|
+
- Colour only on a terminal, and never with `NO_COLOR` set: a log file or
|
|
391
|
+
a pipe gets plain text.
|
|
392
|
+
- An exported "not shown" check counts as shown what replay does: a text
|
|
393
|
+
with invisible format characters in it, a role's name in another case,
|
|
394
|
+
the selected option of a closed dropdown.
|
|
395
|
+
- `saffron eject` says to install `@playwright/test` before
|
|
396
|
+
`npx playwright test`.
|
|
397
|
+
- A message watched for with "and see" (a toast, a flash, a status) is also
|
|
398
|
+
watched inside the page when its target is a test id or plain CSS: one
|
|
399
|
+
shown for a few milliseconds is seen, even on a busy machine or when the
|
|
400
|
+
page moves on at once, where a look every 50 ms could miss it. A watched
|
|
401
|
+
text that holds a secret is never handed to the page.
|
|
402
|
+
- A {unique:...} value the page shows in another case (uppercase through
|
|
403
|
+
CSS) goes back to its token in the recording, so a "not shown" check
|
|
404
|
+
never keeps last run's value; a {unique:name:digits} value the page
|
|
405
|
+
groups ("984,905,203") is refused when the recording is saved, as no
|
|
406
|
+
token stands for it.
|
|
407
|
+
- The check after a heal follows only a tab its action was recorded
|
|
408
|
+
opening, as replay does: a popup a page timer opened mid-check no longer
|
|
409
|
+
passes the check on another page.
|
|
410
|
+
- A secret in a URL's query or fragment typed on the command line
|
|
411
|
+
(`--base-url "https://app.test/?access_token=..."`) is masked in the
|
|
412
|
+
report's command, the HTML report, the console summary and export
|
|
413
|
+
bundles, as a URL password already was.
|
|
414
|
+
- A run whose only reds are scenarios the recorder could not run (a stale
|
|
415
|
+
or missing recording under --no-agent) no longer reads "all clear" in
|
|
416
|
+
the HTML report or "0 failed" alone in `saffron status`: both say the
|
|
417
|
+
run failed.
|
|
418
|
+
- saffron.config.json refuses a number or other non-string for a path or
|
|
419
|
+
name setting (baseURL, features, dataDir, storageState, timezone, model
|
|
420
|
+
and others, and a session's storageState) at load, instead of failing
|
|
421
|
+
later with a raw Node error after agent work.
|
|
422
|
+
- `saffron report --merge` exits 1 when the merged run has a red, or a
|
|
423
|
+
scenario pending review with `strict` on, as `saffron run` does.
|
|
424
|
+
- `saffron diff` with a proposal file that is not there, and `saffron
|
|
425
|
+
author` with a requirements file that is not there, say "<file> does not
|
|
426
|
+
exist" and exit 2 instead of crashing.
|
|
427
|
+
- A placeholder an Outline's Examples have no column for (a misspelt
|
|
428
|
+
`<rol>` beside `| role |`), in a step, doc string or table cell, is now
|
|
429
|
+
an error naming the columns (with inline Examples, only a near miss of a
|
|
430
|
+
column: other tags such as `"<script>"` or `List<String>` stay
|
|
431
|
+
literal), instead of every row running the same text and later rows
|
|
432
|
+
passing on row 1's recording.
|
|
433
|
+
- An Examples block with no table (`Examples: data:roles`, `Examples:
|
|
434
|
+
{roles}`, or a table forgotten) is now an error instead of running no
|
|
435
|
+
row without a word. A header row with no rows yet still runs nothing.
|
|
436
|
+
- A Scenario Outline with no Examples whose steps use a placeholder is
|
|
437
|
+
now refused ("add Examples or use Scenario:") instead of running once
|
|
438
|
+
with its placeholders as literal text.
|
|
439
|
+
- A `*` step opening a scenario after a Background `Given` is no longer
|
|
440
|
+
counted as part of the Background's setup.
|
|
441
|
+
- A .feature file whose description or doc string has a line starting
|
|
442
|
+
`Fixture:` is no longer told to rename to .saffron. Only a `Fixture:`
|
|
443
|
+
line with steps under it is.
|
|
444
|
+
- Editor snippets escape `$`, `\` and `}` in step texts and StepSet names,
|
|
445
|
+
so a name holding one completes as written.
|
|
446
|
+
- The AGENTS.md block `saffron init` writes names the features and data
|
|
447
|
+
folders `saffron.config.json` sets, not always `features/` and `data/`.
|
|
448
|
+
- `saffron init` reads a VS Code `.vscode/mcp.json` with comments and
|
|
449
|
+
trailing commas (JSONC) instead of calling it invalid JSON; one already
|
|
450
|
+
holding the saffron entry is left as is, and one that needs it says what
|
|
451
|
+
to add by hand rather than drop the comments.
|
|
452
|
+
- A database check written as count(*) and checked as 0 is now treated as
|
|
453
|
+
"not in the database": an empty bind is refused, the 0 must keep holding
|
|
454
|
+
for all of db.timeout, and a schema repair of it is refused.
|
|
455
|
+
- A schema repair of a database check whose row count comes from an
|
|
456
|
+
Outline value (a count of `<n>`) is refused, since that value may be 0.
|
|
457
|
+
- On MySQL, a check that locks rows (FOR UPDATE, FOR SHARE, SKIP LOCKED,
|
|
458
|
+
LOCK IN SHARE MODE) is refused.
|
|
459
|
+
- Database writes through a SQL Server synonym or view, or a MySQL view,
|
|
460
|
+
are refused, since they may reach a table in another database.
|
|
461
|
+
- On MySQL, INSERT ... SET is read with its columns, as INSERT ... VALUES
|
|
462
|
+
is.
|
|
463
|
+
- A token in an API response that no step extracts (an access_token or
|
|
464
|
+
refresh_token field, a Location carrying #access_token=) no longer
|
|
465
|
+
reaches the transcript, reports, traces, error lines or the recording
|
|
466
|
+
agent in clear.
|
|
467
|
+
- A polled request (until) keeps polling after a lost connection (a
|
|
468
|
+
reset) or an attempt that timed out, and fails with that error only once
|
|
469
|
+
the poll's time is up. A stopped run, a DNS failure or a TLS failure
|
|
470
|
+
ends the poll at once.
|
|
471
|
+
- The recording agent can no longer record a response-time limit (maxMs)
|
|
472
|
+
or a poll limit (until.timeoutMs) other than the one the scenario states
|
|
473
|
+
in figures: the bound must be a duration the step or the checks right
|
|
474
|
+
after it name, in any unit.
|
|
475
|
+
- A Then whose only check is the 2xx class, or the very status its
|
|
476
|
+
request already required, is refused as a check that could never fail:
|
|
477
|
+
the agent records a field or schema check instead.
|
|
478
|
+
- Two requests at once in a new API session no longer open two clients,
|
|
479
|
+
one of which was never closed.
|
|
480
|
+
- In a parallel run with the agent, a scenario whose replay failed and
|
|
481
|
+
then passed when run again for the agent is red, as with one worker: it
|
|
482
|
+
was reported green, and a failure that comes and goes passed with
|
|
483
|
+
nothing in the report.
|
|
484
|
+
- `saffron run --plan` refuses an invalid `--workers` or
|
|
485
|
+
`--agent-workers`, as a run does.
|
|
486
|
+
- A failed check on a field named as a credential (a token, a session id)
|
|
487
|
+
says `[credential, not kept]` instead of its value, in the error, the
|
|
488
|
+
report and a poll's last reason.
|
|
489
|
+
- Hiding a credential field in a response keeps the rest of the body as
|
|
490
|
+
the API wrote it: large ids, decimals like 1.10 and spacing are not
|
|
491
|
+
rewritten.
|
|
492
|
+
- Exported response checks wait for the page's requests to settle and fail
|
|
493
|
+
when the same request is answered again with a status the check rejects,
|
|
494
|
+
as replay does.
|
|
495
|
+
- Exported "URL does not contain", count and differs checks hold only once
|
|
496
|
+
the page settles, as replay's do: a navigation on its way, a list still
|
|
497
|
+
growing or a value that changes for a moment no longer passes them.
|
|
498
|
+
- Exported captures and differs checks read the displayed copy of an
|
|
499
|
+
element after a hidden one with the same text, as replay does.
|
|
500
|
+
- Exported value checks require a shown input that holds the value in the
|
|
501
|
+
same look.
|
|
502
|
+
- Exported "not shown" checks on a literal name pattern also find the name
|
|
503
|
+
with invisible format characters in it, as replay does.
|
|
504
|
+
- The exported helper file writes the project root with forward slashes on
|
|
505
|
+
Windows, so it reads on macOS and Linux too.
|
|
506
|
+
- A heal of a step that failed partway through is told which of the step's
|
|
507
|
+
actions replay already performed, and records only the rest; a healed
|
|
508
|
+
recording no longer repeats them (an Add click run twice before
|
|
509
|
+
Proceed).
|
|
510
|
+
- Behavior change: a heal that changes nothing in the recording no longer
|
|
511
|
+
files a proposal when its proof passes. Such a run used to be a yellow
|
|
512
|
+
proposal and is now red, saying the recording failed on replay and then
|
|
513
|
+
passed again (timing or the app's state), as it already was with
|
|
514
|
+
--no-verify.
|
|
515
|
+
- A proposal not filed on this machine no longer turns a failed committed
|
|
516
|
+
recording yellow when it checks the Then steps differently from the
|
|
517
|
+
committed recording: the run is red and names the pending file.
|
|
518
|
+
- A check in a browser session that has no page yet (As "Bob") is checked
|
|
519
|
+
on that session's own page, as replay does, never on the default
|
|
520
|
+
session's page.
|
|
521
|
+
- A seeded step that an agent segment went past and recorded nothing for
|
|
522
|
+
keeps its seeded recording instead of an empty one.
|
|
523
|
+
- "Checks proven sensitive" is claimed only when every Then was probed; a
|
|
524
|
+
Then the sensitivity check leaves out (one that clicks, for example)
|
|
525
|
+
leaves it unclaimed.
|
|
526
|
+
- A row value checked through XPath `normalize-space(...)`, `string(...)`
|
|
527
|
+
or `./text()`, or through an unquoted CSS attribute value
|
|
528
|
+
(`[data-k*=x-<id>]`), is kept whole, so row 1 no longer passes or clicks
|
|
529
|
+
on "Order 12". A row value inside another XPath function, such as
|
|
530
|
+
`translate` or `concat`, is refused instead of matched loosely.
|
|
531
|
+
- A "not shown" check whose selector holds the text itself
|
|
532
|
+
(`:has-text(...)`, `text=...`, XPath `contains(., ...)`) fails when the
|
|
533
|
+
page shows that text with invisible characters around the row value,
|
|
534
|
+
such as the bidi isolates Fluent and ICU add.
|
|
535
|
+
- An invisible character (soft hyphen, zero-width space, bidi isolate) no
|
|
536
|
+
longer counts as a word boundary, so "saved" does not pass on "unsaved"
|
|
537
|
+
written with a soft hyphen, and row 1 does not pass on "Order 12" with a
|
|
538
|
+
zero-width space inside.
|
|
539
|
+
- A recorded response, attribute, style or JSON field pattern that
|
|
540
|
+
backtracks fails the step at once with a message saying so, instead of
|
|
541
|
+
freezing the run.
|
|
542
|
+
- A "not shown" check treats straight and typographic quotes alike, so
|
|
543
|
+
"Can't save" fails while the page shows "Can’t save".
|
|
544
|
+
- Exported Playwright tests fail a recorded pattern that backtracks (a
|
|
545
|
+
URL, a response body or JSON field, an attribute, a style) saying so, as
|
|
546
|
+
replay does, no longer take an invisible format character as a word
|
|
547
|
+
boundary, and read straight and typographic quotes alike in "not shown"
|
|
548
|
+
checks.
|
|
549
|
+
- A credential in an API base URL's query (`api.baseURL` or `--api-url`
|
|
550
|
+
with `?api_key=`) is masked in request errors, the transcript and what
|
|
551
|
+
the recording agent reads, as the URL's password already was; one in the
|
|
552
|
+
browser `baseURL` query (`?token=`) is masked in traces too.
|
|
553
|
+
- A failed header check never quotes a credential header's value
|
|
554
|
+
(Set-Cookie) and masks a token in a Location it quotes.
|
|
555
|
+
- A row value inside an XPath `or` (`contains(., "Order <id>") or
|
|
556
|
+
contains(., "Order #<id>")`) is refused as one replay cannot tie to its
|
|
557
|
+
row, as inside `translate` or `concat`: a "not shown" check passed on
|
|
558
|
+
the other branch.
|
|
559
|
+
- A "not shown" check through `:text(...)`, `:text-is(...)`,
|
|
560
|
+
`:text-matches(...)`, XPath `normalize-space(.)="..."` or
|
|
561
|
+
`text()="..."`, or an exact `internal:text="..."s` also finds the text
|
|
562
|
+
with invisible characters around the row value.
|
|
563
|
+
- A numeric one-time code or PIN in an API response (`"otp": 123456`) is
|
|
564
|
+
hidden in the transcript and error lines, as a text one is.
|
|
565
|
+
- A recorded pattern bounds a row value before `.*`, `[^,]*` or `.{0,9}`
|
|
566
|
+
and more text: `/Order <id>.*Shipped/` for row 1 passed on "Order 12
|
|
567
|
+
Shipped". An optional or loose point after the value (`<id>\.?`,
|
|
568
|
+
`<id>[,;]`) no longer lets "1.5" or "1,000" pass for 1.
|
|
569
|
+
- "Not shown" finds text in markup with no space between its elements
|
|
570
|
+
(React, Vue, minified HTML): "Order 1" passed as gone while the row read
|
|
571
|
+
"Order 1Pending", and "Bread" while it read "BreadIn stock". A "not
|
|
572
|
+
shown" by role and name on a role whose name never comes from its text (a
|
|
573
|
+
list item, a status, an alert, a paragraph) reads its text: it passed
|
|
574
|
+
with the text still shown.
|
|
575
|
+
- An attribute check on `checked`, `selected` or `value` reads the control
|
|
576
|
+
as it is now, in replay and in exported tests: a box the app unticked
|
|
577
|
+
passed as ticked.
|
|
578
|
+
- Patterns, captures and amounts never read across a block's edge, as text
|
|
579
|
+
checks and exported tests do: "Qty 1" over "0 left" no longer reads "Qty
|
|
580
|
+
10", nor "$1" over "5 nights" the amount 15. A price split into flex items
|
|
581
|
+
("$" "12" ".50") still reads "$12.50".
|
|
582
|
+
- A row value written unquoted inside an XPath function
|
|
583
|
+
(`contains(@data-order, <id>)`) is refused: row 1 found order 12.
|
|
584
|
+
- A row's number is never written as `<column>` inside a longer number
|
|
585
|
+
("9.5" in "€9.50", "104" in "order-000104"): another row's "not shown"
|
|
586
|
+
looked for "€12.250" and passed. A zero-padded id in a locator now binds
|
|
587
|
+
the recording to its row, so other rows are red until the step names the
|
|
588
|
+
id as the page writes it.
|
|
589
|
+
- A two-column table with a repeated first cell is judged whole, so an
|
|
590
|
+
edit to an earlier row re-records and seeds no other scenario.
|
|
591
|
+
- A long number an Outline row's page sets in its own way ("+4712345678"
|
|
592
|
+
shown "+47 123 45 678", an IBAN in blocks) binds the recording to its
|
|
593
|
+
row, with guidance, where another row passed on it.
|
|
594
|
+
- The recording agent cannot check a value against one captured in the same
|
|
595
|
+
Then, nor a response by a pattern every URL matches, nor bind a database
|
|
596
|
+
check to part of a row's value ("ann" of ann@example.test). A watch armed
|
|
597
|
+
in a step replay ran before the session can be checked.
|
|
598
|
+
- A recording the post-process refuses is told to the agent and ends as an
|
|
599
|
+
ordinary red, its cost counted, not as the provider failing.
|
|
600
|
+
- An API request whose path segment comes out blank (`/orders/{captured:id}`
|
|
601
|
+
with an empty id) is refused, in replay and in exported tests: it went to
|
|
602
|
+
the collection URL and passed. A credential a response echoes under a
|
|
603
|
+
plain name is kept from other origins.
|
|
604
|
+
- Database checks refuse MySQL `||` (run as OR), a WITH that selects a bind
|
|
605
|
+
itself, and on PostgreSQL a backslash before a quote; "at most 0" on a
|
|
606
|
+
count is an absence check, as "equals 0" is.
|
|
607
|
+
- A kept trace no longer holds the saved login (`storageState`), basic-auth
|
|
608
|
+
credentials or extra headers, and every cookie in its network log is
|
|
609
|
+
redacted whatever its name. A `{refresh, access}` token pair's access
|
|
610
|
+
token is a credential: hidden from the agent, the transcript and check
|
|
611
|
+
failures, and kept from other origins.
|
|
612
|
+
- A pinned contract applies `required` on a `readOnly` property to
|
|
613
|
+
responses only and on a `writeOnly` one to requests only, so one shared
|
|
614
|
+
model no longer refuses every request body. A response the contract
|
|
615
|
+
gives as `*/*` is checked against its schema.
|
|
616
|
+
- `saffron status`, `saffron coverage` and the editors keep the last run's
|
|
617
|
+
result of each scenario when the report came from a CI machine on another
|
|
618
|
+
OS.
|
|
619
|
+
- `--rerecord` puts back a recording it removed when the scenario was not
|
|
620
|
+
recorded again (the app down, a capability off, the budget spent,
|
|
621
|
+
Ctrl-C). `run --plan --rerecord` replays a fixture, as the run does.
|
|
622
|
+
- `saffron report --merge --strict` exits 1 on a pending review, as shards
|
|
623
|
+
run with `--strict` do.
|
|
624
|
+
- A parallel run's kept red no longer names evidence its second try
|
|
625
|
+
deleted.
|
|
626
|
+
- A feature edit that would drop an Outline `<column>` is skipped, and a
|
|
627
|
+
file with mixed line endings keeps every other line's ending.
|
|
628
|
+
- A short config key of the team's own (`app`, `id`) is ignored, not
|
|
629
|
+
refused as a misspelling; a short key's letters swapped (`evn`) still is.
|
|
630
|
+
- `differs from` compares `$5` and `5 USD` as one currency.
|
|
631
|
+
|
|
632
|
+
Editors (VS Code extension 0.8.2, JetBrains plugin 0.8.2)
|
|
633
|
+
|
|
634
|
+
- Database steps: a step recorded with a database check or write is not
|
|
635
|
+
offered as a zero-token wording to reuse (completion, the step hover and
|
|
636
|
+
StepSet expansion say it replays at zero tokens only in its own
|
|
637
|
+
scenario), and a step is marked as one only when every recording of it
|
|
638
|
+
reads or writes the database.
|
|
639
|
+
- A proposal with the runner's new "database check repaired after a schema
|
|
640
|
+
change:" or "database write added by the heal:" note is held from Accept
|
|
641
|
+
All, as `saffron accept --all` skips it.
|
|
642
|
+
- The last run's line, a CI run's totals and its rows count a red the app
|
|
643
|
+
did not cause (a stale recording without the agent, a missing
|
|
644
|
+
environment, the agent or the budget) as not run, as the runner's status
|
|
645
|
+
does.
|
|
646
|
+
- Open trace finds a CI scenario whose feature path the runner left out,
|
|
647
|
+
by its name.
|
|
648
|
+
- Eject's This file runs in the open file's own Saffron project, with
|
|
649
|
+
forward slashes on Windows, and is not offered for a file outside every
|
|
650
|
+
project.
|
|
651
|
+
- VS Code: a status or diff that times out is stopped with its child
|
|
652
|
+
processes, a cut diff says so, and `{data:...}` hints read the
|
|
653
|
+
environment as the runner does. JetBrains: the Dashboard reloads only on
|
|
654
|
+
new runner output, and proposals and lenses no longer read the disk on
|
|
655
|
+
the UI thread.
|
|
656
|
+
|
|
657
|
+
## 1.3.5: Scenarios kept from running at once
|
|
658
|
+
|
|
659
|
+
Released 2026-11-08.
|
|
660
|
+
|
|
661
|
+
Runner
|
|
662
|
+
|
|
663
|
+
- `@serial:<name>` keeps scenarios that would trip over each other's data
|
|
664
|
+
from running at the same time. With `--workers` or `--agent-workers`,
|
|
665
|
+
scenarios that share a name run one at a time, in file order, while the
|
|
666
|
+
rest of the suite runs beside them; a plain `@serial` is one shared name.
|
|
667
|
+
On a Feature line it covers the whole file. `--shard` keeps every
|
|
668
|
+
scenario under one name on the same machine, so two machines never run
|
|
669
|
+
them at once either.
|
|
670
|
+
- A value is no longer found inside a number grouped by spaces or
|
|
671
|
+
apostrophes: `250 kr` passed on `1 250 kr`, and `1 250` on
|
|
672
|
+
`1 250 000`. Nor is an amount found inside another currency written with
|
|
673
|
+
the same symbol: `$5.00` passed on `R$5.00` and `CA$5.00`. Both were
|
|
674
|
+
false greens, in text, pattern and download checks.
|
|
675
|
+
- A response check's row value is no longer found inside a longer word
|
|
676
|
+
written percent-encoded in the URL: `Jos` passed on `/users/Jos%C3%A9`.
|
|
677
|
+
- A captured decimal keeps the digits the API sent: `0.123456789012345678`
|
|
678
|
+
was rounded, so a `differs from {captured:...}` check passed on an
|
|
679
|
+
unchanged value, and a later request sent another number.
|
|
680
|
+
- `should have a non-empty "/id"` holds on a number: a numeric id was red
|
|
681
|
+
on every run.
|
|
682
|
+
- The recording agent no longer sees a base URL's password: the APIs'
|
|
683
|
+
summary in its prompt, a refused request and a request's errors quoted
|
|
684
|
+
it. A secret percent-encoded the way Python writes it (`%28`, `%2A`) is
|
|
685
|
+
masked too.
|
|
686
|
+
- The dashboard, `saffron status` and the CI run panel count reds the app
|
|
687
|
+
did not cause (a stale recording in a run with `--no-agent`) as *not
|
|
688
|
+
run*, as the HTML report does. The dashboard counted one as failed, and
|
|
689
|
+
said it had been red since before its grid. A scenario whose latest
|
|
690
|
+
result is such a red now counts as not run, never as passing on an older
|
|
691
|
+
green: the dashboard said 100% passing after a run that exited 1.
|
|
692
|
+
- "Not shown" no longer passes while the text is on screen beside a
|
|
693
|
+
number: `iPhone 15` was taken for part of `15 128` in
|
|
694
|
+
`iPhone 15 128 GB`. A positive check reads a number as grouped with the
|
|
695
|
+
next only after one to three digits that are not part of a word, so
|
|
696
|
+
`Galaxy S24` is found in `Galaxy S24 256 GB` again.
|
|
697
|
+
- Thai, Lao, Khmer and Myanmar digits keep a number whole (`1` passed on
|
|
698
|
+
`12` written in Myanmar digits), as do the Arabic decimal point and
|
|
699
|
+
thousands separator (`١٫٥`) and a thin space between thousands in a
|
|
700
|
+
downloaded file.
|
|
701
|
+
- A base URL password holding a raw `@` is masked whole: its tail after
|
|
702
|
+
the `@` reached the recording agent.
|
|
703
|
+
- An untyped `differs from` holds a number however the step writes it: an
|
|
704
|
+
unchanged `10.5` passed against `10.50`, and `100` against `1e2`.
|
|
705
|
+
- A captured list or object keeps each number as the API wrote it. A
|
|
706
|
+
decimal JavaScript cannot hold was sent on as a string; it is now
|
|
707
|
+
refused, as the same number captured alone is.
|
|
708
|
+
- `@serial` holds while a heal proves an Outline's other rows: a row with
|
|
709
|
+
its own `@serial` Examples tag was replayed without its name. Tags are
|
|
710
|
+
read in any case, with commas between names (`@Serial:Rooms` and
|
|
711
|
+
`@serial:a,b` were ignored), and `saffron eject` writes `workers: 1`
|
|
712
|
+
for a suite with `@serial` scenarios, or warns when the project has its
|
|
713
|
+
own Playwright config.
|
|
714
|
+
- "Not shown" counts a number beside a comma as shown: `2` passed while
|
|
715
|
+
`Tags: 1,2,3` was on screen, and `42` beside `1001,42`. It also counts
|
|
716
|
+
`$5.00` as shown in `US$5.00`, and a role name pattern with a row value
|
|
717
|
+
reads `15 128` as one number, as a text target does.
|
|
718
|
+
- `256 GB` is found in `Galaxy S24 256 GB`: a model number heads no
|
|
719
|
+
grouped number.
|
|
720
|
+
- The dashboard counts a scenario whose latest run was not run against
|
|
721
|
+
the pass rate: one scenario passing while the rest were not run read
|
|
722
|
+
100%, 1 of 1.
|
|
723
|
+
- A `differs from` checked as a string compares a number too large for
|
|
724
|
+
JavaScript by its digits: the same int64 id passed.
|
|
725
|
+
- A base URL password holding a quote or a space is masked before the
|
|
726
|
+
recording agent reads the base URL.
|
|
727
|
+
- The exported helper refuses `$10.50` for a role name recorded as
|
|
728
|
+
`Refund of $10.`, keeps a captured object's numbers as the API wrote
|
|
729
|
+
them, and `saffron eject` gives the trace warning beside the
|
|
730
|
+
`@serial` one, where it gave only the `@serial` one.
|
|
731
|
+
- An Outline's "not shown" check on a row value the recording glued into
|
|
732
|
+
a word reads the word in any form. Recorded on `apples` as `<item>s`,
|
|
733
|
+
row `box` checked that `boxs` was not shown and passed while `boxes`
|
|
734
|
+
was listed; it now counts `boxes` as shown.
|
|
735
|
+
- An exported "not shown" check looks in the app's own frames, as replay
|
|
736
|
+
does: it passed while `Card declined` showed in a same-origin iframe. It
|
|
737
|
+
also waits for the page's requests and for the hidden state to hold, as
|
|
738
|
+
replay does: it passed before an error drawn 300 ms after a click.
|
|
739
|
+
- The dashboard keeps every browser of the last CI run's matrix, where
|
|
740
|
+
each browser's shards are merged on their own: the browser merged last
|
|
741
|
+
hid the others' reds. A red from before browsers were recorded stands
|
|
742
|
+
when a later run is not run in one browser and passes in another.
|
|
743
|
+
- A `differs from` checked as a string fails on the same number, `true`
|
|
744
|
+
or `null`: it passed on any of them.
|
|
745
|
+
- `saffron eject` run again no longer warns about `@serial` when the
|
|
746
|
+
config it wrote already runs one test at a time.
|
|
747
|
+
- `run_plan` (MCP) and `saffron run --plan` name each `@serial` name and
|
|
748
|
+
how many tests share it, and the agent skill's syntax reference lists the
|
|
749
|
+
tags that change how a scenario runs.
|
|
750
|
+
|
|
8
751
|
## 1.3.4: Checks that wait for a settled page, and credentials kept masked
|
|
9
752
|
|
|
10
753
|
Released 2026-11-07.
|
|
@@ -1512,7 +2255,7 @@ Runner
|
|
|
1512
2255
|
newest marked. A duration over a minute reads as minutes and seconds,
|
|
1513
2256
|
and scenarios sharing a cause get a button for the command that deals
|
|
1514
2257
|
with them all, the command itself still on the page.
|
|
1515
|
-
- **A compatibility page per release.** [Compatibility](COMPATIBILITY.md)
|
|
2258
|
+
- **A compatibility page per release.** [Compatibility](docs/COMPATIBILITY.md)
|
|
1516
2259
|
lists the Node.js, operating systems, Playwright and browsers, each
|
|
1517
2260
|
agent's SDK (the range accepted and the version tested) and the recording
|
|
1518
2261
|
versions this release reads and writes, generated from the release's own
|
|
@@ -1524,7 +2267,7 @@ Runner
|
|
|
1524
2267
|
greens); `--record` also records the suite afresh with the agent and
|
|
1525
2268
|
measures the share verified on the first attempt, the share held for
|
|
1526
2269
|
review, and the time and cost. The results are published in the docs
|
|
1527
|
-
([benchmark](BENCHMARK.md)): today 66 of 66 replays green and no false
|
|
2270
|
+
([benchmark](docs/BENCHMARK.md)): today 66 of 66 replays green and no false
|
|
1528
2271
|
green on 8 broken apps.
|
|
1529
2272
|
- **A coverage map per pull request.** `saffron coverage` maps each
|
|
1530
2273
|
requirement ID (a tag matching `requirementTag`, such as `@REQ-12`) to
|