saffron-ai 1.3.5 → 1.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +1809 -3
- package/README.md +22 -6
- package/dist-pkg/cli.js +475 -432
- package/package.json +49 -4
- package/skills/saffron/SKILL.md +70 -16
- package/skills/saffron/references/api.md +11 -2
- package/skills/saffron/references/config.md +4 -3
- package/skills/saffron/references/database.md +327 -0
- package/skills/saffron/references/syntax.md +51 -14
- package/templates/playwright/saffron-helpers.ts +855 -153
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,1812 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.4.1: Database steps on seven more databases
|
|
9
|
+
|
|
10
|
+
Released 2026-11-17.
|
|
11
|
+
|
|
12
|
+
Runner
|
|
13
|
+
|
|
14
|
+
- Database steps on six more databases (beta), beside MongoDB below, each
|
|
15
|
+
tested against the real database or its emulator:
|
|
16
|
+
- Oracle (`oracle://user:pass@host:1521/service`, `npm install oracledb@6
|
|
17
|
+
node-sql-parser`; the driver's thin mode needs no Oracle client). A
|
|
18
|
+
statement is read as plain SQL: `FETCH FIRST`, `OFFSET` and `ROWNUM`
|
|
19
|
+
count as a `LIMIT`, and `CONNECT BY`, `(+)`, flashback, `q'...'`
|
|
20
|
+
strings, `MERGE` and PL/SQL are refused. Reads run in a `READ ONLY`
|
|
21
|
+
transaction; a write is read back by its primary key, the key Oracle
|
|
22
|
+
gave an inserted row read from that row itself. Names are in capitals,
|
|
23
|
+
so a row's columns are checked as `/0/NAME`.
|
|
24
|
+
- SQLite (`sqlite:path/to/app.db`, `npm install node-sql-parser`; the
|
|
25
|
+
driver is Node's own, from Node 22.13). The file is opened read-only
|
|
26
|
+
for a check and never created. `INSERT OR REPLACE` and the like are
|
|
27
|
+
refused. A "not in the database" check that only a type keeps empty
|
|
28
|
+
(`json_extract(...) = $1` with text where the JSON holds a number) is
|
|
29
|
+
refused. `db.allow` lists the file's name or path.
|
|
30
|
+
- MariaDB (`mariadb://` or `mysql://`). It runs as MySQL, and did not
|
|
31
|
+
before: every statement failed on a session setting MariaDB does not
|
|
32
|
+
have, and cleanup put a `JSON` column back respelt (`1.10` as `1.1`).
|
|
33
|
+
- CockroachDB (`cockroachdb://` or `postgres://`). It runs as PostgreSQL,
|
|
34
|
+
and did not before: every write failed, as CockroachDB does not say
|
|
35
|
+
which table a written row is in. A table with no primary key (a hidden
|
|
36
|
+
`rowid`) counts as having none, and a read the database asks to run
|
|
37
|
+
again is read again.
|
|
38
|
+
- DynamoDB (`dynamodb://<region>`, or `dynamodb://<host>:<port>` for
|
|
39
|
+
DynamoDB Local; `npm install @aws-sdk/client-dynamodb`). A step holds
|
|
40
|
+
one JSON command, as on MongoDB: `get`, `query`, `put` or `update`,
|
|
41
|
+
with no scan and no delete. A key attribute is sent as the type the
|
|
42
|
+
table declares; a "not in the database" check is red when an item is
|
|
43
|
+
found with a bind sent as another type, and never reads a global
|
|
44
|
+
secondary index. A `put` never replaces an item. Tested on DynamoDB
|
|
45
|
+
Local; AWS itself is not part of the test suite.
|
|
46
|
+
- Firestore (`firestore://<project-id>`, or
|
|
47
|
+
`firestore://<host>:<port>/<project-id>` for the emulator; `npm
|
|
48
|
+
install @google-cloud/firestore@7`). One JSON command per step: `get`,
|
|
49
|
+
`query`, `create`, `add` or `update`. Placeholders name Firestore's
|
|
50
|
+
types (`"$1:timestamp"`, `"$1:reference"`); a "not in the database"
|
|
51
|
+
check is red when a document is found with a bind sent as another
|
|
52
|
+
type. A `create` never writes over a document. Tested on the
|
|
53
|
+
emulator; Google Cloud itself is not part of the test suite.
|
|
54
|
+
- What tells AI agents how to use Saffron is brought up to this release:
|
|
55
|
+
the authoring skill and its references (the new databases and how to
|
|
56
|
+
word a step for each, exact headers and keywords, `--filter` with
|
|
57
|
+
`and`, `or` and `not`, amounts compared to the decimals shown), the
|
|
58
|
+
block `saffron init` writes into `AGENTS.md` and `CLAUDE.md`, and the
|
|
59
|
+
MCP server's `project_status`, which names the database's engine and
|
|
60
|
+
how its steps are worded. Run `npx saffron init` again to take the new
|
|
61
|
+
block and skill.
|
|
62
|
+
- A recording with a database step on one of the new engines is written
|
|
63
|
+
as schema version 22, so an older runner asks for an update instead of
|
|
64
|
+
a re-record it could not make.
|
|
65
|
+
- Database steps on MongoDB (beta): set `db.url` to a `mongodb://` or
|
|
66
|
+
`mongodb+srv://` URL that names its database, and `npm install mongodb`
|
|
67
|
+
(no SQL parser needed). A step holds one JSON command where a SQL step
|
|
68
|
+
holds its statement: `{"find": "customers", "filter": {"email": "$1"}}`
|
|
69
|
+
or `{"count": ...}` for a check, `{"insert": ..., "documents": [...]}`
|
|
70
|
+
or `{"update": ..., "filter": {...}, "set": {...}}` for a write. The
|
|
71
|
+
rules are the SQL ones: every value a placeholder bound from the step
|
|
72
|
+
(typed where the field needs it: `"$1:objectId"`, `"$1:uuid"`,
|
|
73
|
+
`"$1:long"`, `"$1:decimal"`, `"$1:date"`), a check tied to a value the
|
|
74
|
+
scenario wrote, filters limited to comparison operators (`$where`,
|
|
75
|
+
`$expr`, `$regex` and aggregation are refused), and cleanup that deletes
|
|
76
|
+
inserted documents and puts back, type for type, only the paths an
|
|
77
|
+
update set. A "not in the database" check is judged again with each
|
|
78
|
+
bind sent as each other type it reads as (text, ObjectId, UUID, number,
|
|
79
|
+
decimal, long, date), at its start and at the end of its window: a
|
|
80
|
+
document found so is red, as is a field no document has (compared with
|
|
81
|
+
a bind, in a range, under `$elemMatch` or `$exists` true) or a
|
|
82
|
+
collection that is not there. A UUID reads as its dashed text. A
|
|
83
|
+
whole document or list compared as a value is refused (compare its
|
|
84
|
+
fields with dotted paths), and so is a projection beside a check that a
|
|
85
|
+
field is missing. An update that fails part way puts back what it
|
|
86
|
+
changed first. A failed MongoDB check is never repaired. Recordings
|
|
87
|
+
with MongoDB steps are cache version 21, so an older runner asks to be
|
|
88
|
+
updated instead of advising a re-record it cannot do.
|
|
89
|
+
- On every engine, an `every` check under `LIMIT`, `TOP` or `DISTINCT`
|
|
90
|
+
is refused, as a count is: it cannot see a second, wrong row.
|
|
91
|
+
- A value extracted from inside a credential object (`/session/id`,
|
|
92
|
+
`/session/0/id`, `/credentials/access`, kept as `ref`) is now a
|
|
93
|
+
credential too when it is a text of 6 characters or more: it is masked,
|
|
94
|
+
and never sent to another origin, as the body already showed it hidden
|
|
95
|
+
(`/auth/role` holding "admin" stays plain). So is a credential inside an
|
|
96
|
+
extracted object (`extract "/data"` holding `session: { id }`), a list
|
|
97
|
+
under such an object (`auth.backup[0]`), the texts of a list or object
|
|
98
|
+
whose name ends in `tokens` (`refreshTokens`, while `total_tokens`
|
|
99
|
+
holding a number stays plain), and a name in the plural (`apiKeys`,
|
|
100
|
+
`passwords`). Before, the
|
|
101
|
+
session id or token could reach a second host. Exported tests read
|
|
102
|
+
credentials the same way.
|
|
103
|
+
- Postgres: `db.cleanup` now puts back a json or jsonb column whose change
|
|
104
|
+
was only past 15 digits (`9007199254740992` to `9007199254740993`).
|
|
105
|
+
Both sides of the comparison were read as JavaScript numbers, so the
|
|
106
|
+
change looked like none and the new value stayed. A write's `extract`
|
|
107
|
+
of such a number now keeps every digit instead of a neighbouring id.
|
|
108
|
+
- An API scenario's database checks now read again at `pollIntervalMs`,
|
|
109
|
+
as a browser scenario's do, instead of every 250 ms.
|
|
110
|
+
- An interaction that ran is no longer run again by `retries`: an action
|
|
111
|
+
recorded as opening a tab when none opens, a click armed for a download
|
|
112
|
+
or a file chooser that starts none, a click or key press whose page load
|
|
113
|
+
outlasts the action timeout, and a checkbox the click did not change all
|
|
114
|
+
fail at once. Before, a retry clicked again, and a submit could send its
|
|
115
|
+
order twice (and, on a reloaded form, pass). A `newTab` whose page fails
|
|
116
|
+
to load is closed, so a retry's tab is the only one.
|
|
117
|
+
- A response header sent more than once is checked as it means: each
|
|
118
|
+
Set-Cookie on its own (of several cookies, a pattern names its cookie
|
|
119
|
+
first, `^session=.*HttpOnly`), and a header that holds one value
|
|
120
|
+
(X-Frame-Options, Content-Type, Location) with every value, so a `DENY`
|
|
121
|
+
beside an `ALLOWALL` is red. A list-valued header (Vary, Cache-Control)
|
|
122
|
+
is still read joined. Exported tests check headers the same way.
|
|
123
|
+
- A JSON check whose path holds an empty row value (`/stock/<id>` with an
|
|
124
|
+
empty cell) is refused instead of reading a field no response has, where
|
|
125
|
+
"not there" passed on anything. A check typed as a string fails on a big
|
|
126
|
+
number, so an id the API turned from text into a number is red.
|
|
127
|
+
- Recordings that may turn red with this release, and need
|
|
128
|
+
`saffron run --rerecord`: a check typed as a string on a big number
|
|
129
|
+
(the old advice for such numbers; check it with no type), and a
|
|
130
|
+
Set-Cookie pattern that names no cookie on a response that sets several.
|
|
131
|
+
- Credentials in a form-encoded answer (an OAuth token endpoint's
|
|
132
|
+
`access_token=...&scope=...`) and in JSON after an anti-XSSI prefix are
|
|
133
|
+
hidden from the agent, errors and transcripts.
|
|
134
|
+
- An Examples table with a header row and no rows is now an error. Before,
|
|
135
|
+
the outline ran nothing, so deleting its last row removed it from the
|
|
136
|
+
run while CI stayed green. Comment a draft out until it has a row.
|
|
137
|
+
- `--max-session-minutes` (`maxSessionMinutes`) now limits API recording
|
|
138
|
+
and healing sessions too, and they print their progress as browser ones
|
|
139
|
+
do.
|
|
140
|
+
- A header pattern (`expectHeader`) or a downloaded file's pattern that
|
|
141
|
+
backtracks (`(a+)+$`) now fails the check instead of freezing the run,
|
|
142
|
+
as JSON and page patterns already did.
|
|
143
|
+
- A JSON data file with a number a JavaScript number reads as another
|
|
144
|
+
value (an id past 2^53, more digits than a double holds, `1e400` read
|
|
145
|
+
as Infinity or `1e-400` as 0) is now an error that asks for it in
|
|
146
|
+
quotes, instead of sending and checking another value.
|
|
147
|
+
- A feature file whose scenario header Gherkin does not know (`scenario:`,
|
|
148
|
+
`Scenario :`, `Scenario Name` with no colon, another language's keyword
|
|
149
|
+
under an English file) is now an error. Gherkin reads such a line, and
|
|
150
|
+
the steps under it, as the feature's description, so the scenario, or
|
|
151
|
+
the whole file, left the run without a word while the suite stayed
|
|
152
|
+
green. So is a step above a scenario's first step that Gherkin cannot
|
|
153
|
+
read as one (a tab or a no-break space after its keyword, an invisible
|
|
154
|
+
character before it), a `StepSet` call there in a `.feature` file, and
|
|
155
|
+
`stepset` in the wrong case: each was left out of its scenario. Prose is
|
|
156
|
+
left alone: one description line that starts with "When", or a wrapped
|
|
157
|
+
line that starts with "and", stays description. The language server
|
|
158
|
+
reports these, and an Examples table with no rows or no table, in every
|
|
159
|
+
editor that uses it.
|
|
160
|
+
- A recording is recorded again when a step reads a table value another
|
|
161
|
+
step gave it (`<table:name>`) and an edit leaves two tables disagreeing
|
|
162
|
+
on that value. Before, the token resolved to nothing, replay looked for
|
|
163
|
+
the text `<table:name>` itself, and a "not shown" check passed on it
|
|
164
|
+
with the value on the page.
|
|
165
|
+
- A "not shown" check ending in a row's whole number (`<amt>` holding 10)
|
|
166
|
+
now counts `10.00` on the page as shown. It passed as gone.
|
|
167
|
+
- A "not shown" check in a step that names no column, holding a word of
|
|
168
|
+
the row's value in another order or spelling ("Lee, Ann" for the guest
|
|
169
|
+
"Ann Lee"), now binds its recording to that row, so the other rows fail
|
|
170
|
+
with guidance instead of asking whether the first row's guest is gone.
|
|
171
|
+
- An amount check compares to the unit the amount is shown with: a rate
|
|
172
|
+
of 0.125 no longer passes on 0.129. Amounts shown with two decimals or
|
|
173
|
+
fewer still compare to the cent, so a tax worked out as 7.9992 is the
|
|
174
|
+
8.00 shown. Two captures that differ under a cent (0.125 and 0.129) now
|
|
175
|
+
count as differing. The unit is read from the text shown, so a rate
|
|
176
|
+
shown as 0.1200 does not pass on 0.1204, nor 0.0000002 on 0.0000001.
|
|
177
|
+
- `saffron accept --all` now holds a `Then` recorded again with another
|
|
178
|
+
pattern (`^Paid$` become `^Pending$` or `^.*$`), another attribute or
|
|
179
|
+
style property, another response URL pattern, a URL check turned from
|
|
180
|
+
"contains" to "excludes", or another compared capture, under "check
|
|
181
|
+
weakened:". Only values, statuses, URLs and JSON checks were compared.
|
|
182
|
+
The same pattern with an anchor gained, or grouped another way, is not
|
|
183
|
+
held.
|
|
184
|
+
- Database checks: a query whose WHERE picks one of its table's own rows
|
|
185
|
+
by a subquery (`WHERE id = (SELECT max(id) FROM orders WHERE ref = $1)`,
|
|
186
|
+
`created_at >= ALL (...)`) is read as cutting the rows down, as `LIMIT 1`
|
|
187
|
+
is: a count of the rows or "no second row" under it is refused, and a
|
|
188
|
+
repair may not add it. A subquery on another table is left alone.
|
|
189
|
+
- Database checks: a "not in the database" check refuses a bind of spaces
|
|
190
|
+
alone, as an empty one, and a bind that is a whole number past 2^53,
|
|
191
|
+
which a recording had already rounded to a neighbouring id (bind it as
|
|
192
|
+
text).
|
|
193
|
+
- Database rows shown in a failed check and in the report hide a column
|
|
194
|
+
named as a credential (`api_token`, `password_hash`), as a response
|
|
195
|
+
body's fields are hidden. `SELECT *` on a users table put its hashes and
|
|
196
|
+
tokens in the report. A name ending in `hash` or `digest` after a
|
|
197
|
+
credential's (`passwordHash`, `token_digest`) now names a credential
|
|
198
|
+
everywhere.
|
|
199
|
+
- A repair of a database check bound to a number an earlier step kept
|
|
200
|
+
(an id) is no longer refused: the recorded bind carries its type, which
|
|
201
|
+
a repair cannot write, so the two never compared equal. The recorded
|
|
202
|
+
bind is the one kept, and the one the repair is checked with on the
|
|
203
|
+
live data before its proposal is filed.
|
|
204
|
+
- MongoDB, beyond the first entry above: a whole document or list compared under
|
|
205
|
+
`$eq`, `$ne`, `$in`, `$nin` or `$all` is refused, as it is written
|
|
206
|
+
plainly; a projection or a set path through `$`, `$[]` or `$[name]` is
|
|
207
|
+
refused (name a list's item by its number, `items.0.qty`); `$0` is no
|
|
208
|
+
placeholder; a `:date` that is no day of the calendar (`2026-02-30`,
|
|
209
|
+
hour 24) is refused instead of read as the next day. A "not in the
|
|
210
|
+
database" check is judged on each bound field alone as well, so two
|
|
211
|
+
binds sent as the wrong type at once no longer pass, and a check that a
|
|
212
|
+
field is missing is refused under a projection at any depth. A write
|
|
213
|
+
whose documents cannot be read back is undone at once (inserted
|
|
214
|
+
documents deleted, set paths put back) instead of left for a cleanup
|
|
215
|
+
that never heard of it. An update that set an item past the end of a
|
|
216
|
+
list puts the list back whole, also when it changed an item of that
|
|
217
|
+
list in the same update, and a path named as an object's own
|
|
218
|
+
member (`constructor`) is removed again.
|
|
219
|
+
- MySQL: options in `db.url`'s query can no longer turn the adapter's own
|
|
220
|
+
off. `?multipleStatements=true` or `?decimalNumbers=true` in a URL
|
|
221
|
+
shared with the app won over them.
|
|
222
|
+
- Exported tests hide a credential field in a response body they quote
|
|
223
|
+
and never quote a credential header's value, and read a token pair's
|
|
224
|
+
access token as a credential, all as replay does.
|
|
225
|
+
- The reset commands (`reset` in the config) no longer run before a
|
|
226
|
+
pending proposal is only replayed: every later Examples row, and a
|
|
227
|
+
scenario waiting for review, reset the project's data and then recorded
|
|
228
|
+
nothing. They run when a recording really follows.
|
|
229
|
+
- `saffron author` with the Antigravity provider no longer fails at "the
|
|
230
|
+
session's turn limit" after the draft came back.
|
|
231
|
+
- In a heal, an assert, a watch or a network wait the agent makes before
|
|
232
|
+
its first step, or under a step that had run, is answered as not
|
|
233
|
+
recorded, with why. It was answered "Assertion recorded.", and surfaced
|
|
234
|
+
only as a refused finish.
|
|
235
|
+
- `saffron diff --html <file>` no longer writes the review page over the
|
|
236
|
+
file named after it. `--html` takes its file only as `.html`: a proposal
|
|
237
|
+
or feature file there is a file to review, and the page goes to its
|
|
238
|
+
default place. Before, a recording that cost AI usage, or a feature
|
|
239
|
+
file, was replaced by the page.
|
|
240
|
+
- `saffron report --merge` folds the last report's rows in only for
|
|
241
|
+
shards of a run that named its scenarios (`file:line`, `file:name`).
|
|
242
|
+
Shards of a `--filter` or path run were read the same way, so
|
|
243
|
+
`latest.json` held another run's scenarios and reds.
|
|
244
|
+
- `saffron status` (and the dashboard and the editors' panels) reads a
|
|
245
|
+
last report whose rows are not rows as no report, instead of crashing.
|
|
246
|
+
- `saffron eject` refreshes its own specs when they have CRLF line
|
|
247
|
+
endings (a Windows checkout), instead of keeping them for good as
|
|
248
|
+
edited.
|
|
249
|
+
- `saffron accept --all` and `reject --all` exit 1, saying it once, when
|
|
250
|
+
a pending proposal cannot be read.
|
|
251
|
+
- Config: `actionTimeoutMs: 0` is refused (Playwright reads 0 as no
|
|
252
|
+
timeout; here every check timed out at once), and so is a `transport`
|
|
253
|
+
that is not an object or `capabilities` given as a list.
|
|
254
|
+
- `--filter` refuses `&&`, `!` and `~` from other tools instead of reading
|
|
255
|
+
them as tags: `"@a && @b"` ran `@a` or `@b`.
|
|
256
|
+
- An Outline with thousands of rows (an Examples data file) no longer
|
|
257
|
+
costs memory by the square of its rows: 20,000 rows ran out of memory.
|
|
258
|
+
- Found in a further review, before release:
|
|
259
|
+
- A "not shown" check, and a check that the URL does not hold a text,
|
|
260
|
+
passed on the page a click was about to leave (a redirect a moment
|
|
261
|
+
after the click, to a page that answers slowly). The moment a passing
|
|
262
|
+
state must hold is now timed between two looks, and the URL is read
|
|
263
|
+
once a navigation on its way has arrived.
|
|
264
|
+
- A Set-Cookie pattern is run only on the cookie it names. Written with
|
|
265
|
+
no `^`, or with a second branch (`^id=.*Secure|HttpOnly`), it held on
|
|
266
|
+
another cookie. Exported tests check it the same way.
|
|
267
|
+
- An Outline with several Examples blocks: a `<column>` that only
|
|
268
|
+
another block has is an error. Before, the block without it replayed
|
|
269
|
+
with the text `<column>` as written.
|
|
270
|
+
- "The price differs from before" no longer passes when the element
|
|
271
|
+
stopped showing a price ("Price unavailable"): a price against a text
|
|
272
|
+
that is no amount is refused. Exported tests refuse it the same way.
|
|
273
|
+
- A row value followed by a point or comma at the end of a selector's
|
|
274
|
+
text (`"Qty <n>."`) is kept whole: row 1 no longer finds `Qty 1.5 kg`.
|
|
275
|
+
A minus written as another hyphen (U+2010, U+2011) is read as one.
|
|
276
|
+
- "Not shown" finds text the page writes in the other Unicode form (`é`
|
|
277
|
+
as `e` and an accent).
|
|
278
|
+
- A click whose handler keeps the page busy past the action timeout is
|
|
279
|
+
not clicked again by `retries`.
|
|
280
|
+
- Oracle: text is bound as `CHAR`, so it matches a `CHAR` or `NCHAR`
|
|
281
|
+
column as a literal does. Before, `code = $1` found no row of a
|
|
282
|
+
`CHAR(10)` column, and "not in the database" passed with the row
|
|
283
|
+
there. A `NUMBER` reads as a number whatever its decimals (`12.5`
|
|
284
|
+
read as text, `12` as a number, so no one check served both rows of
|
|
285
|
+
an Outline); only one past 15 digits reads as text. `SELECT UNIQUE`,
|
|
286
|
+
`NEXTVAL` in a check, and the `UTL_` and `DBMS_` packages are refused.
|
|
287
|
+
- Database checks: more aggregates are read as the one row they always
|
|
288
|
+
return (`json_group_array`, `total`, `median`, `jsonb_agg_strict`,
|
|
289
|
+
`corr`, `regr_*`, `range_agg`, `APPROX_COUNT_DISTINCT` and the like),
|
|
290
|
+
so a count or an exists on that row is refused.
|
|
291
|
+
- SQLite: a write to a table declared `ON CONFLICT REPLACE` is refused
|
|
292
|
+
while cleanup is on, as it deletes the row in its way for good. A
|
|
293
|
+
"not in the database" check that sends `true` or `false` as text
|
|
294
|
+
against a boolean is refused, as a number sent as text is.
|
|
295
|
+
- MariaDB: `/*M! ... */` comments, which MariaDB runs, are refused as
|
|
296
|
+
`/*! ... */` is.
|
|
297
|
+
- DynamoDB and Firestore: a "not in the database" check is red on an
|
|
298
|
+
attribute no item has that the filter needs to exist (`$exists`
|
|
299
|
+
true), and on a set or a list compared with a value by equality.
|
|
300
|
+
- MongoDB: a "not in the database" check is red on a field no document
|
|
301
|
+
has in a range, `$exists` true or `$elemMatch` too; a document found
|
|
302
|
+
as the bind is sent is the check's own red, with no type blamed; a
|
|
303
|
+
seed under a number `_id` past int32 no longer fails after writing
|
|
304
|
+
its document; `"$1:uuid"` binds a UUID, which reads as its dashed
|
|
305
|
+
text; and a suite whose MongoDB is down waits 2 seconds for each
|
|
306
|
+
scenario after the first, not 10.
|
|
307
|
+
- A response body's credentials stay hidden when a masked number left
|
|
308
|
+
the body no JSON (`"id": {env:USER_ID}`); before, every token in that
|
|
309
|
+
response stayed in the transcript. A credential repeated inside
|
|
310
|
+
another text (a link, a hint) or in JSON kept as text is hidden too.
|
|
311
|
+
`auth_key`, `authKey` and `X-Auth-Key` are read as credentials, and a
|
|
312
|
+
key in a base URL's query no longer reaches the agent's prompt.
|
|
313
|
+
- A contract's `pattern` that only JavaScript's unicode flag refuses
|
|
314
|
+
(`[\w-\.]`, `\_`) no longer makes every schema check fail to compile,
|
|
315
|
+
and an operation is found by a path the URL encodes (`/price list/{id}`).
|
|
316
|
+
- Masking no longer takes seconds on one long run of letters joined by
|
|
317
|
+
`-` or `.`, and a request with a large body is masked once, not per
|
|
318
|
+
field.
|
|
319
|
+
- A repaired database check's proposal is replayed on later runs and
|
|
320
|
+
on an Outline's other rows. Before, the agent repaired it again on
|
|
321
|
+
every run, and the other rows went red.
|
|
322
|
+
- A pending heal proposal is replayed once the failed replay's own
|
|
323
|
+
database rows are undone, so a `Given` that seeds a row under a
|
|
324
|
+
literal unique value no longer makes it "not replay" and heal again
|
|
325
|
+
on every run.
|
|
326
|
+
- A heal whose proof marks an action as opening a tab files a proposal
|
|
327
|
+
bound to the committed recording as its file holds it. Before, it was
|
|
328
|
+
stale the moment it was filed: `saffron accept` refused it and the
|
|
329
|
+
next run healed again.
|
|
330
|
+
- A heal that changed nothing shows the step the committed recording
|
|
331
|
+
failed on, and names it in the error. Before, every step read as
|
|
332
|
+
passed on a red.
|
|
333
|
+
- A proof of refined steps that cannot open stops the scenario on that
|
|
334
|
+
error. Before, the refined steps were filed with the first proof's
|
|
335
|
+
failure.
|
|
336
|
+
- A browser that dies while a recording is set up is no longer reported
|
|
337
|
+
as the AI provider's fault ("not run: provider").
|
|
338
|
+
- `saffron accept --all` holds more checks recorded again looser: a
|
|
339
|
+
longer time limit, another header, schema or JSON field of a group, a
|
|
340
|
+
group check that gained a filter, another bind or query of a database
|
|
341
|
+
check, and a URL pattern that lost its `^` or `$`.
|
|
342
|
+
- A scenario with no name is an error. It was recorded, then its
|
|
343
|
+
recording and proposal could never be read.
|
|
344
|
+
- A fixture's proposal can apply its feature edits
|
|
345
|
+
(`accept --with-feature-edit`).
|
|
346
|
+
- A URL holding a row's value as the browser writes it
|
|
347
|
+
(`Smith,%20John`) is recorded as its `<column>`, so each row checks
|
|
348
|
+
its own; a recording that kept the recorded row's value there is
|
|
349
|
+
recorded again.
|
|
350
|
+
- The MCP `verify` tool runs one call at a time: two at once could
|
|
351
|
+
answer with each other's scenarios. The MCP and language servers
|
|
352
|
+
follow a `.env` that changed while they run.
|
|
353
|
+
- `run --plan` no longer says a MongoDB, DynamoDB or Firestore insert is
|
|
354
|
+
kept under `db.cleanup: "inserts"`.
|
|
355
|
+
- `saffron coverage --fail-on-loss` (or `--markdown`) with no `--base`,
|
|
356
|
+
and `saffron report --strict` with no `--merge`, exit 2 instead of
|
|
357
|
+
passing over the flag and exiting 0.
|
|
358
|
+
- `saffron init` leaves a `CLAUDE.md` that only reads `AGENTS.md` in
|
|
359
|
+
(`@AGENTS.md`) as it is, where it wrote a second copy of the block,
|
|
360
|
+
and adds to a `.gitignore` in the file's own line endings.
|
|
361
|
+
- Found in a third review, before release:
|
|
362
|
+
- Ctrl-C while the agent records starts no new agent session, ends the
|
|
363
|
+
one running, and files no proposal for the scenario it was on. A run
|
|
364
|
+
stopped by its terminal closing (SIGHUP) stops as Ctrl-C does: it
|
|
365
|
+
writes its report, runs `teardown` and exits 129.
|
|
366
|
+
- `--rerecord` sets each recording aside instead of deleting it, and
|
|
367
|
+
puts it back when the run is killed before recording it again. The
|
|
368
|
+
next run puts back what a killed run left aside, unless a recording
|
|
369
|
+
or proposal for that scenario was written since.
|
|
370
|
+
- A fixture the replay phase ran is not run again when a scenario of
|
|
371
|
+
its feature goes on to the agent.
|
|
372
|
+
- Two runs writing `.saffron/reports/latest.json` at once no longer
|
|
373
|
+
lose each other's scenarios.
|
|
374
|
+
- `saffron import` refuses a bundle whose evidence or proposal would be
|
|
375
|
+
written through a symbolic link out of `.saffron`.
|
|
376
|
+
- Recording: a response field or header checked against the value that
|
|
377
|
+
same response gave (extracted in the When) is refused, as it holds
|
|
378
|
+
whatever the API answered; so is a header checked against a value
|
|
379
|
+
extracted in the same Then. A group's status check names a status
|
|
380
|
+
the step names, as a status check does.
|
|
381
|
+
- Recording: an `{env:...}`, `{data:...}` or kept value that is also a
|
|
382
|
+
column's name is never written inside its `<column>` placeholder,
|
|
383
|
+
where replay typed the literal text `<user>`. A `{data:...}` value
|
|
384
|
+
inside a password no longer leaves the rest of the password in the
|
|
385
|
+
recording. A capture whose name holds an env value is renamed with
|
|
386
|
+
its references, and a request's extract names are renamed too.
|
|
387
|
+
- Recording: a kept value of tens of thousands of characters no longer
|
|
388
|
+
makes every later request fail to record. An action no saved
|
|
389
|
+
recording could load is refused when recorded: a capture named `""`,
|
|
390
|
+
a blank text check (an empty field is checked by its value), a
|
|
391
|
+
status written `" 404"`. A status class written `4XX` is kept as
|
|
392
|
+
`4xx`. A row value a JSON check's pattern holds escaped is its
|
|
393
|
+
column, as in other patterns. A time limit written in words ("within
|
|
394
|
+
two seconds") is the bound it names.
|
|
395
|
+
- Exported tests check as replay does: a pattern or capture never reads
|
|
396
|
+
across a block's edge ("Qty 1" over "0 left" is not "Qty 10"), a
|
|
397
|
+
download holds the value as written, never across a line, "not
|
|
398
|
+
shown" reads words as replay reads them, a role's name in another
|
|
399
|
+
case counts as shown and an option a closed dropdown does not show
|
|
400
|
+
does not, credentials in a response are hidden and a URL's secrets masked as
|
|
401
|
+
replay does, a captured credential encoded in a URL goes to the APIs
|
|
402
|
+
under test only, and a multipart request never sends
|
|
403
|
+
`saffron.config.json`. A check on a target recorded outside an
|
|
404
|
+
iframe is exported as stricter, as the export looks for it in the
|
|
405
|
+
page only. `saffron eject` warns that `setup`, `teardown` and `reset`
|
|
406
|
+
are not run by the exported tests, and a data list the data does not
|
|
407
|
+
hold refuses only the scenario that reads it.
|
|
408
|
+
- Database checks: `count(*)` reads as text on PostgreSQL, MySQL,
|
|
409
|
+
MariaDB and CockroachDB, and as a number on SQL Server, Oracle and
|
|
410
|
+
SQLite, as the guide now says. Oracle refuses more network and
|
|
411
|
+
scheduling packages (`UTL_URL`, `DBMS_LDAP`, `HTTPURITYPE` and the
|
|
412
|
+
like). SQLite's refusal of `true` or `false` sent as text says to
|
|
413
|
+
send 1 or 0.
|
|
414
|
+
- `saffron accept --all` holds a check recorded again without its
|
|
415
|
+
contract check, or on another request group.
|
|
416
|
+
- Found in a fourth review, before release:
|
|
417
|
+
- A replay that failed, then passed when the recording was replayed
|
|
418
|
+
again for a heal, is red with a note that it failed on replay and then
|
|
419
|
+
passed, where it was green. Without an agent it is not replayed again.
|
|
420
|
+
- Database checks: a "not in the database" check is judged again with
|
|
421
|
+
its binds as the other types the database may keep them as, on every
|
|
422
|
+
engine: on MongoDB, DynamoDB and Firestore a range's two ends and the
|
|
423
|
+
values of `$all` together; on PostgreSQL, CockroachDB, MySQL, MariaDB
|
|
424
|
+
and SQLite, text that reads as a number, `true` or `false` as 1 or 0,
|
|
425
|
+
and the values inside a JSON bind. A `count(*)` checked as 0 on SQLite
|
|
426
|
+
and an "at most" count are judged too. Each held with the row there.
|
|
427
|
+
- MongoDB: `$elemMatch` on a field held as one value is refused in a
|
|
428
|
+
"not in the database" check; so are, on DynamoDB and Firestore, a list
|
|
429
|
+
or set compared by `in`, `array-contains`, `$in` or `$beginsWith`, and
|
|
430
|
+
one value compared by `array-contains`. Each held whatever the item
|
|
431
|
+
held.
|
|
432
|
+
- MongoDB, DynamoDB and Firestore take a `:binary` bind (`0x` hex), so
|
|
433
|
+
a field held as bytes can be checked. MongoDB and DynamoDB take a null
|
|
434
|
+
bind, which finds a field held as null, where it was refused and the
|
|
435
|
+
agent was told to write `$exists: false`, which never finds one.
|
|
436
|
+
- DynamoDB: a probe value DynamoDB will not compare (BETWEEN a number
|
|
437
|
+
and a text) no longer fails a valid check. With `db.url` naming a
|
|
438
|
+
region, an endpoint set in the environment (`AWS_ENDPOINT_URL_DYNAMODB`)
|
|
439
|
+
or a profile is no longer used. On DynamoDB and Firestore, a read that
|
|
440
|
+
runs past its time on a database that answers is the check's red,
|
|
441
|
+
never an outage.
|
|
442
|
+
- SQL: in a "not in the database" check, a `LIKE` bind holding a
|
|
443
|
+
backslash (a `[` on SQL Server) is refused, as it never finds the text
|
|
444
|
+
it holds. `$0` is refused. The latest row written as an anti-join, a
|
|
445
|
+
`max()` read through a derived table, and SQL Server's `FOR JSON` and
|
|
446
|
+
`FOR XML` count as keeping one row. On SQL Server, an alias that is a
|
|
447
|
+
word a statement starts with (`SHUTDOWN`) is refused, and a date
|
|
448
|
+
written as text reads year, month, day whatever the login's language.
|
|
449
|
+
Oracle refuses every `UTL_` and `DBMS_` package called (`DBMS_LOB`
|
|
450
|
+
aside) and `APEX_WEB_SERVICE`, as the guide says.
|
|
451
|
+
- MySQL and MariaDB: an `UPDATE` that changed rows its `WHERE`, read
|
|
452
|
+
first, did not find (a `COLLATE` the snapshot lost) is rolled back,
|
|
453
|
+
where cleanup left them changed; a write waiting on a table's lock
|
|
454
|
+
gives up at the step's time, where it waited a year; an `UPDATE` of a
|
|
455
|
+
`FLOAT` under cleanup is refused, as the driver reads it to 6 digits.
|
|
456
|
+
- Schema repairs: a `count(*)`, `LIMIT` or `DISTINCT` check can be
|
|
457
|
+
repaired (each repair was red); a repair of a step's second or later
|
|
458
|
+
check is taken from the repaired check alone, as the agent is told to
|
|
459
|
+
record it; a check that expects no rows, which no repair could be held
|
|
460
|
+
to, starts no agent session.
|
|
461
|
+
- A recording made again (`--rerecord`, or a stale one) whose step now
|
|
462
|
+
writes to the database is held for review against the one it replaces,
|
|
463
|
+
and a first recording whose When writes a table a later Then reads is
|
|
464
|
+
held too. Its note keeps the name `database write added by the heal:`,
|
|
465
|
+
so Accept All in an editor holds it as before. The block `saffron init`
|
|
466
|
+
writes into `AGENTS.md` says so to agents: a `When` acts through the
|
|
467
|
+
app, never the database.
|
|
468
|
+
- Recording: a Then that checked the very field the When had typed into,
|
|
469
|
+
for the value typed and with no page loaded since, passed whether or
|
|
470
|
+
not the app saved it: it is refused, and the agent checks where the
|
|
471
|
+
app shows the saved value, or the field once the page loads again. An
|
|
472
|
+
item or every check that compared a list's entries with a value taken
|
|
473
|
+
from that same list is refused, as the same check on the taken field
|
|
474
|
+
was. A database check bound to a `{unique:...}` no Given or When
|
|
475
|
+
writes, a `{data:...}` no step writes, or a `<table:key>` no table
|
|
476
|
+
holds is refused (a misspelt `{unique:email}` held "not in the
|
|
477
|
+
database" for good).
|
|
478
|
+
- Recording: a time limit in words or in parts is the bound it names:
|
|
479
|
+
"twenty-five seconds" was read as 5 s, "1 minute 30 seconds" as 60 s,
|
|
480
|
+
and "within a second" or "half a minute" as no limit. A response header
|
|
481
|
+
kept as `Header:Location` is kept as `header:Location`, which the
|
|
482
|
+
recording format loads. A group check filtered by a status the step
|
|
483
|
+
does not spell out is recorded: only a status a check expects must be
|
|
484
|
+
named.
|
|
485
|
+
- Recording: a tab closed or opened under a Then was recorded and not
|
|
486
|
+
held, so "the payment window closes" passed when the agent closed it:
|
|
487
|
+
it is held for review, as a click there is. A value of tens of
|
|
488
|
+
thousands of characters (an upload's base64, a long `{data:...}` text)
|
|
489
|
+
made recording fail with "Regular expression too large" after its
|
|
490
|
+
request was sent: it is compared as it stands. Two writes a step asks
|
|
491
|
+
for to one operation were kept as one when they sent a form, a
|
|
492
|
+
multipart, or JSON bodies with other fields (`{sku, quantity}`, then
|
|
493
|
+
`{sku, amount}`): only a field renamed, retyped or added to the same
|
|
494
|
+
values marks a corrected attempt. A capture named with a space whose
|
|
495
|
+
name held an `{env:...}` value had its references garbled: they follow
|
|
496
|
+
its new name. An Outline's database or response field check that
|
|
497
|
+
states a value of the row an earlier step wrote, read through
|
|
498
|
+
`{captured:...}`, follows the row, where every other row was refused.
|
|
499
|
+
- Browser checks: a text check read the items of a flex row, a grid or
|
|
500
|
+
floated boxes a space apart, so "$15" passed on a price drawn as `$15`
|
|
501
|
+
`.99`, and "$50.00" on `-` `$50.00`: a number that goes on across items
|
|
502
|
+
reads as one, and "$12.50" holds on `$` `12` `.50`. Text drawn with no
|
|
503
|
+
area (font-size 0, or in a box scaled to nothing such as a closed
|
|
504
|
+
menu's `transform: scaleY(0)`) counted as shown, so "I see Delete
|
|
505
|
+
account" and "Delete account is not shown" both passed on one page: it
|
|
506
|
+
counts as not shown, and the failure says why. A page rewriting its
|
|
507
|
+
history every few hundred milliseconds kept every settled check from
|
|
508
|
+
ever passing: only a navigation to another document starts the hold
|
|
509
|
+
again. Exported tests read the same.
|
|
510
|
+
- An Outline row value before a point in an XPath function replay
|
|
511
|
+
cannot tie (`contains(translate(., ...), 'qty <n>.')`) is refused, as
|
|
512
|
+
it was without the point, where row 1 passed on "Qty 1.5 kg". An
|
|
513
|
+
Outline whose database or API check names a column but follows nothing
|
|
514
|
+
of it is refused in that check's own words, where it was told to show
|
|
515
|
+
the value on the page.
|
|
516
|
+
- `saffron accept --all` holds a Then recorded again on a wider element
|
|
517
|
+
when only its selectors, its role or its frame changed (a total read on
|
|
518
|
+
`body`, a "not shown" that lost its payment frame). A hand-edited
|
|
519
|
+
recording with an HTTP field on a browser action is refused by name,
|
|
520
|
+
where the staleness check crashed on it.
|
|
521
|
+
- API steps: a request carried an `{env:...}` value or a captured
|
|
522
|
+
credential to another host when it came inside a captured value (a
|
|
523
|
+
`Location` that echoed `?key=` back on another host, captured and
|
|
524
|
+
followed): it is refused, in replay and in exported tests. An exported
|
|
525
|
+
test sent a captured credential written as a query value (`s.urlValue`)
|
|
526
|
+
to another host: it is refused, as replay refuses it. A Set-Cookie
|
|
527
|
+
pattern with an unescaped dot in the cookie's name ran on another
|
|
528
|
+
cookie the dot also matched: the name is read as written. A schema
|
|
529
|
+
check passed on a JSON body served as `text/plain`, a type the
|
|
530
|
+
contract does not give that response: it is red, naming the type.
|
|
531
|
+
- `run --plan` and the MCP `run_plan` tool plan a stale recording with a
|
|
532
|
+
pending proposal as the proposal replay the run makes, where they said
|
|
533
|
+
"record again (AI)". A `baseURL` or `--base-url` that is not an
|
|
534
|
+
`http://` or `https://` URL is refused, naming the key, where every
|
|
535
|
+
API scenario failed with "Invalid URL". `saffron import` refuses to
|
|
536
|
+
write through a `.saffron`, `proposals` or `artifacts` folder that is
|
|
537
|
+
itself a symbolic link.
|
|
538
|
+
- The language server marks a `{unique:...}` that is no token
|
|
539
|
+
(`{unique:e mail}`, `{unique:}`) where it is written, as `saffron run`
|
|
540
|
+
refuses to start for it; JetBrains showed nothing until the run.
|
|
541
|
+
- Found while publishing, and in the review that followed:
|
|
542
|
+
- Playwright colours its error log wherever `TERM`, `COLORTERM`,
|
|
543
|
+
`FORCE_COLOR` or the CI vendor (GitLab CI, CircleCI) give colours, also
|
|
544
|
+
into a pipe or a log file and with `NO_COLOR` set. GitHub Actions and a
|
|
545
|
+
shell without `TERM` get none, which is where this project's tests ran.
|
|
546
|
+
Saffron read that text as it came:
|
|
547
|
+
- The rules that keep an interaction from being sent twice matched
|
|
548
|
+
nothing with the codes in it, so a click whose handler kept the page
|
|
549
|
+
busy past the action timeout was clicked again by the retry.
|
|
550
|
+
- A typed `{env:...}` value that holds a line break (a PEM key, a JSON
|
|
551
|
+
service account) was written in clear to the HTML and JSON reports
|
|
552
|
+
when its fill failed, and was shown to the recording agent, in the
|
|
553
|
+
error's own text where its line breaks stand as typed: the codes
|
|
554
|
+
inside the value kept it from being masked there. A value of 3 to 5
|
|
555
|
+
characters right after a code was left as well.
|
|
556
|
+
- Every failed step's error carried the codes into reports and logs.
|
|
557
|
+
|
|
558
|
+
Playwright's colours are now turned off for the run, and every error is
|
|
559
|
+
read, masked and reported without the codes. The tests run with colours
|
|
560
|
+
on.
|
|
561
|
+
- Known limit, as in 1.4.0: a typed `{env:...}` value of several lines (a
|
|
562
|
+
PEM key, a pasted JSON key file) is masked where its line breaks stand
|
|
563
|
+
as typed, JSON-escaped, percent-encoded or inside base64, and not where
|
|
564
|
+
they are written another way: as single spaces, as a page shows the
|
|
565
|
+
text and the recording agent's snapshot reads it; as CR LF, as a form
|
|
566
|
+
post sends it (`%0D%0A` in a form body); as `↵`, in Playwright's call
|
|
567
|
+
log. There it reaches the agent, the terminal, the reports, the history
|
|
568
|
+
and the trace. Do not type a secret of several lines into a field in a
|
|
569
|
+
scenario. A one-line value with a tab or two spaces in a row is not
|
|
570
|
+
masked where a page shows one space there.
|
|
571
|
+
- A value inside base64 was missed when other text was glued before the
|
|
572
|
+
encoding by `/`, `-` or `_` (`/v/<base64>` in a path, `sess-<base64>`
|
|
573
|
+
in an id), or when the encoding's `+`, `/` or `=` were written `%2B`,
|
|
574
|
+
`%2F` or `%3D` in a query string: the trace kept it and the recording
|
|
575
|
+
agent saw it. Such a run is decoded from after each of those characters
|
|
576
|
+
too. Also in 1.4.0.
|
|
577
|
+
- Outline rows: a check that something is no longer shown, in a step that
|
|
578
|
+
does not name the date column, that held the recorded row's day by its
|
|
579
|
+
month's name without the year ("Nov 20", "20 November", "Fri, Nov 20"
|
|
580
|
+
for a cell `2026-11-20` or `{date+5}`) was bound to no row. Every other
|
|
581
|
+
row then asked whether row 1's day was gone, which holds whatever became
|
|
582
|
+
of its own entry. Such a check binds the recording to its row: the other
|
|
583
|
+
rows are refused, with the step to change. A recording made before this
|
|
584
|
+
is recorded once more on the first row with another day. Also in 1.4.0.
|
|
585
|
+
- A database check that a value "differs" passed on the unchanged value
|
|
586
|
+
when the database reads the column as text (a PostgreSQL `numeric`,
|
|
587
|
+
`bigint` or `count(*)`, a MySQL `DECIMAL` or `BIGINT`): `12.50` was
|
|
588
|
+
never the text `12.5`. Text that spells a number is now compared as the
|
|
589
|
+
number, with no type or with `"type": "number"`; `"type": "string"`
|
|
590
|
+
compares the text. A "differs" check typed as a number or a boolean on a
|
|
591
|
+
value of another type is refused, since it held whatever the row held.
|
|
592
|
+
A text column that holds digits is compared as numbers too (`1.10` is
|
|
593
|
+
"the same" as `1.1`, `5` as `5.0`), except a code written with a leading
|
|
594
|
+
zero on either side (a zip `1234` differs from `01234`): where the text
|
|
595
|
+
itself is what matters, give the check `"type": "string"`. Also in
|
|
596
|
+
1.4.0.
|
|
597
|
+
- An "at most" database check (`count(*) AS n` with `max`) passed with any
|
|
598
|
+
number of rows there when its bind was empty or null (an Outline row
|
|
599
|
+
with an empty cell, an empty capture): the count was 0. It is refused,
|
|
600
|
+
as a "not in the database" check is. To count rows whose column is
|
|
601
|
+
rightly empty, compare its length with the number 0
|
|
602
|
+
(`length(note) = $1`: run on PostgreSQL, MySQL, MariaDB and SQLite; on
|
|
603
|
+
Oracle an empty text is NULL, so write `IS NULL`, beside a column
|
|
604
|
+
compared with a bind). Also in 1.4.0. The
|
|
605
|
+
refusal names the way out of its own engine: it named
|
|
606
|
+
`length(note) = $1` on Oracle too, where an empty text and its length
|
|
607
|
+
are NULL, so the check it led the recording agent to counted no row and
|
|
608
|
+
held whatever the table held. On Oracle it says `IS NULL` beside a
|
|
609
|
+
bound column, on SQL
|
|
610
|
+
Server and MongoDB it names no function they do not have, and a null
|
|
611
|
+
bind is called the "at most" check it is.
|
|
612
|
+
- MySQL and MariaDB: a "not in the database" or "at most" check that
|
|
613
|
+
binds an unquoted `true` where the rows are found by its text (a
|
|
614
|
+
`BOOLEAN` or number column holding false or 0, which MySQL reads the
|
|
615
|
+
text as, or a text column holding 'true') was told only that the
|
|
616
|
+
column holds the step's value as another type. It is told what to
|
|
617
|
+
send: 1 or 0 for a `BOOLEAN` or number column (a `CAST` of the column
|
|
618
|
+
to text holds there too), the quoted text for a text one.
|
|
619
|
+
Such a check on a `BOOLEAN` or number column whose row holds false or
|
|
620
|
+
0 is refused though it is right, where 1.4.0 passed it on MySQL: bind 1
|
|
621
|
+
or 0.
|
|
622
|
+
- `npm install saffron-ai` no longer fails with ERESOLVE in a project
|
|
623
|
+
whose own app depends on another version of a database driver (MongoDB
|
|
624
|
+
6, Firestore 8 or 9, Oracle 5 or 26, a pinned AWS SDK, mssql 11, mysql2
|
|
625
|
+
3.11): the optional drivers accept any version at install, and the
|
|
626
|
+
version is checked where a database step loads the driver. Another
|
|
627
|
+
major version than the tests run is refused with the engine, the major
|
|
628
|
+
needed and the version found ("database steps on MongoDB need mongodb
|
|
629
|
+
7; this project has 6.22.0: npm install mongodb@7"). The messages for a
|
|
630
|
+
missing driver name the version to install.
|
|
631
|
+
- An interaction that reached the page is never sent again by a retry: a
|
|
632
|
+
select, a fill, a hover, a drag, an upload, and a key press (a captured
|
|
633
|
+
credential's own included) whose handler kept the page busy past the
|
|
634
|
+
action timeout ran it twice (a dropdown that submits its form sent it
|
|
635
|
+
twice, a drop was dropped twice). It fails as it is, as a click does. So
|
|
636
|
+
does a drag whose drop target came only after the timeout, the mouse
|
|
637
|
+
being down already. An element that is only covered, disabled, hidden,
|
|
638
|
+
read-only or not there yet is still tried again. A step that passed only
|
|
639
|
+
on such a second try is now red: give the page more time with
|
|
640
|
+
`actionTimeoutMs`.
|
|
641
|
+
- A "not in the database" check, and an "at most" check, no longer pass
|
|
642
|
+
with the row there when the read that judges them (the same query with a
|
|
643
|
+
bind sent as another type) fails for a reason that is not the type: the
|
|
644
|
+
app's lock on a row only that read reaches, a timeout, a deadlock or a
|
|
645
|
+
lost connection. Only the database refusing the value as that type counts
|
|
646
|
+
as "not kept as that type"; a lock is read again and is red if held to
|
|
647
|
+
the end of `db.timeout`, and any other error fails the check. Seen on SQL
|
|
648
|
+
Server with a bytes column and a join.
|
|
649
|
+
- An exported Playwright test printed the `{env:...}` value it typed, the
|
|
650
|
+
element it named or the URL it opened when that action failed (a
|
|
651
|
+
disabled or missing field), in the terminal and the CI log, with the
|
|
652
|
+
line, list, dot, json and junit reporters too. Exported actions on such
|
|
653
|
+
a value, or on a captured credential, now run through the helper's
|
|
654
|
+
`masked`, as the checks do, and name the token. Export again to get it.
|
|
655
|
+
(Also in 1.4.0.)
|
|
656
|
+
- A pattern check of an API response, a header or a database row that
|
|
657
|
+
holds `{unique:<column>}` (which recording writes for an Outline) was
|
|
658
|
+
refused on every row with "A regex quantifier cannot contain a row or
|
|
659
|
+
data token": each row is checked against its own generated value.
|
|
660
|
+
- Known cost of the wait a check makes for its action's answer (new in
|
|
661
|
+
this release, below): a page whose latest action left a request open (a
|
|
662
|
+
long-poll or a stream started on load, with the checks straight after
|
|
663
|
+
the page load) makes each of those checks wait half the action timeout,
|
|
664
|
+
3 seconds at most, and hold for half a second: four checks take about
|
|
665
|
+
12 seconds. In replay the checks after the next interaction do not wait
|
|
666
|
+
for it. In an exported test a count, "not shown", "differs", "URL does
|
|
667
|
+
not contain" or response check on such a page waits that long wherever
|
|
668
|
+
it stands, as in 1.4.0, and an exported test of nine checks that wait
|
|
669
|
+
fails on Playwright Test's 30 second test timeout. An answer slower
|
|
670
|
+
than that wait is not waited
|
|
671
|
+
for by a single check, whatever `actionTimeoutMs` is: write the wait
|
|
672
|
+
into the step ("When I pay and wait for the payment API to return
|
|
673
|
+
402"), then check what the answer shows before what it must leave as it
|
|
674
|
+
was.
|
|
675
|
+
- A key press with no target ends at the action's timeout: on a page whose
|
|
676
|
+
key handler never returns the run hung with no verdict and no report. It
|
|
677
|
+
fails saying the page did not answer, and the key is not pressed again.
|
|
678
|
+
- A run whose output goes away (`saffron run | head`, a pager quit) no
|
|
679
|
+
longer dies with exit 1 and a stack
|
|
680
|
+
trace before its teardown and report: it goes on without output, runs
|
|
681
|
+
`teardown`, writes its report and exits with its own verdict. Known
|
|
682
|
+
limit: when the error stream is gone too (`saffron run 2>&1 | head`), a
|
|
683
|
+
`setup`, `reset` or `teardown` command that prints fails on its own
|
|
684
|
+
write, so a
|
|
685
|
+
`setup` that prints stops the run and `echo done && cleanup` does not
|
|
686
|
+
reach the cleanup. Send such a hook's output to a file.
|
|
687
|
+
- With `--browser webkit`, an app that is not running is counted as
|
|
688
|
+
blocked by the environment, as in Chromium and Firefox. It was counted
|
|
689
|
+
as a failure in the app, and with an agent on, a heal session started
|
|
690
|
+
for every scenario.
|
|
691
|
+
- Database outages said as they are: SQL Server with a certificate the
|
|
692
|
+
machine does not trust names the certificate and
|
|
693
|
+
`?trustServerCertificate=true`, where it said the database did not
|
|
694
|
+
answer; an Oracle host that does not resolve is the database not
|
|
695
|
+
answering; a missing SQLite file under a folder with a bracket in its
|
|
696
|
+
name is the environment; an error the app's own trigger raises that says
|
|
697
|
+
"Connection lost" is the statement failing.
|
|
698
|
+
- `saffron runs`, `import --run`, `report --run` and `trace <run>` no
|
|
699
|
+
longer fail when the shell exports `CLICOLOR_FORCE=1` or `GH_FORCE_TTY`:
|
|
700
|
+
gh and az are run with colours and the forced terminal off.
|
|
701
|
+
- `saffron steps` marks a divergent step `◐`, where it shared `●` with a
|
|
702
|
+
recorded one and differed only by a colour that a pipe, a log file and
|
|
703
|
+
an AI agent's shell never get. The language server's hover does too.
|
|
704
|
+
- `saffron init` reads end of input at its question about the examples
|
|
705
|
+
(Ctrl-D, a closed input) as "no" and goes on, where it ended with exit 0
|
|
706
|
+
and nothing set up; it no longer asks when `CI` is set; it says
|
|
707
|
+
`updated` for a `.gitignore` that was already there.
|
|
708
|
+
- `saffron mcp` typed into a terminal says it is a server for an AI
|
|
709
|
+
assistant and how to add it, and exits 2, where it printed nothing and
|
|
710
|
+
waited. The `--filter` refusal no longer ends with two full stops, and a
|
|
711
|
+
recording's target with nothing to find it by is told
|
|
712
|
+
`fallbackSelectors`, the field's name.
|
|
713
|
+
- Playwright 1.65.1 (from 1.64.0; the range is now `^1.65.1`). Run
|
|
714
|
+
`npx playwright install` after upgrading: it brings Chromium 159,
|
|
715
|
+
Firefox 159 and WebKit 27.2. A project with its own `@playwright/test`
|
|
716
|
+
or `playwright` updates that to the same version first
|
|
717
|
+
(`npm install -D @playwright/test@1.65.1`, or
|
|
718
|
+
`npm install -D playwright@1.65.1` where the project depends on
|
|
719
|
+
`playwright`): beside an older one npm
|
|
720
|
+
installs a second Playwright for Saffron, `npx playwright install`
|
|
721
|
+
installs the browsers of the project's own, and the run goes on
|
|
722
|
+
saying a browser is missing. Its trace no longer says which load state a
|
|
723
|
+
wait waited for, so the trace view names it "Wait for load state".
|
|
724
|
+
Playwright 1.65 itself waits for an element inside an `inert` part of
|
|
725
|
+
the page to stop being inert before acting, leaves text the browser does
|
|
726
|
+
not draw (a closed `<details>`) out of accessible names, and matches
|
|
727
|
+
cookies to paths as RFC 6265 says (`Path=/api` no longer goes to
|
|
728
|
+
`/apiv2`): a recording that relied on the old behaviour turns red. The
|
|
729
|
+
recording agent's browser still names elements as Playwright 1.62 does,
|
|
730
|
+
so a new recording can hold a name with such undrawn text too: its
|
|
731
|
+
action or positive check is red on replay, and a "not shown" check on
|
|
732
|
+
that name passes whatever the page shows. Target such an element by a
|
|
733
|
+
test id or a selector.
|
|
734
|
+
- The release gate (`npm run gate`, which `npm publish` runs): the unit
|
|
735
|
+
tests, every database engine one file at a time (it fails where Docker
|
|
736
|
+
does not answer, where it skipped them all), then both end-to-end
|
|
737
|
+
scripts on the bundle that is published.
|
|
738
|
+
- Found in a fifth review, before release:
|
|
739
|
+
- Database checks: a "not in the database" check held whatever the
|
|
740
|
+
database held when the value was kept as a type its bind was never tried
|
|
741
|
+
as: a JSON string, for a bind the statement reads as JSON
|
|
742
|
+
(`data->'ref' = $1`, a `jsonb` column, MySQL's `CAST($1 AS JSON)`), as
|
|
743
|
+
the text 4471 read as JSON is the number; a JSON boolean, for a MySQL
|
|
744
|
+
bind of 1 or 0, as MySQL compares a JSON true with TRUE and never with
|
|
745
|
+
1; and bytes, on every SQL engine, for the `0x` hex a row shows them as.
|
|
746
|
+
Each is tried now, so such a check is red, saying what found the rows
|
|
747
|
+
and how to compare: as text (`data->>'ref' = $1`), `true` or `false`
|
|
748
|
+
unquoted for a JSON boolean (a bind of "true" was told to send 1 or 0)
|
|
749
|
+
and 1 or 0 for a `BOOLEAN` column, the column's hex on that engine. A
|
|
750
|
+
bind the SQL makes JSON of itself (`to_jsonb(CAST($1 AS text))`,
|
|
751
|
+
`JSON_QUOTE($1)`) is refused in such a check.
|
|
752
|
+
- Database checks: a bind only a negation compares (`<> $2`, `NOT IN`,
|
|
753
|
+
`$ne`, `$nin`, `!=`, `not-in`) made a right "not in the database" check
|
|
754
|
+
red, saying to send that bind as another type, which would have broken
|
|
755
|
+
it: it is left as sent, since sent as another type it finds more rows,
|
|
756
|
+
never fewer. On Firestore a query that finds documents as sent blames no
|
|
757
|
+
type for them. A type probe that met the app's lock on the rows (a busy
|
|
758
|
+
SQLite file) failed the check at once: the check reads again, as its
|
|
759
|
+
read loop does.
|
|
760
|
+
- Database checks: a check that a column is not in a row (`/0/discount`
|
|
761
|
+
exists false) on a SQL engine held whatever the column held once the
|
|
762
|
+
`SELECT` left it out or misspelt it, and failed whatever it held once
|
|
763
|
+
the `SELECT` named it: it is refused, saying to write
|
|
764
|
+
`discount IS NULL`; a field inside a JSON column is still checked so. On
|
|
765
|
+
MongoDB, DynamoDB and Firestore a field checked as not in a document is
|
|
766
|
+
red when no document of the collection holds it (a misspelt name), as a
|
|
767
|
+
filter's field already was.
|
|
768
|
+
- Database checks: a count, or a check that there is no second row, under
|
|
769
|
+
a `WHERE` that keeps one row of each group
|
|
770
|
+
(`id IN (SELECT max(id) FROM orders GROUP BY ref)`, a
|
|
771
|
+
`row_number() OVER (...)` column compared in the `WHERE`) passed on a
|
|
772
|
+
double submit: it is refused, as under `LIMIT`. A JSON number past 2^53
|
|
773
|
+
(an external id in a `json`, `jsonb` or Oracle `JSON` column) read
|
|
774
|
+
rounded, so "differs" passed on the unchanged value and "equals" was red
|
|
775
|
+
on the right one: it reads as its digits, as MySQL reads it, and
|
|
776
|
+
Oracle's cleanup, which put it back rounded, puts the JSON back as it
|
|
777
|
+
was.
|
|
778
|
+
- Database checks: the repair of a step's second database check after a
|
|
779
|
+
schema change was refused before any session when an earlier check of
|
|
780
|
+
the step, which still runs as recorded, expects no rows or was recorded
|
|
781
|
+
without its row count: only the check that failed and the checks the
|
|
782
|
+
repair changes are held to those rules. A recording whose `Then` holds a
|
|
783
|
+
database write (a hand edit, a merge) loaded and ran it, so the `Then`
|
|
784
|
+
checked what it had just written: it is refused, and so is a database
|
|
785
|
+
check under a `Given` or `When`.
|
|
786
|
+
- Database steps, while recording: during a heal the agent could seed the
|
|
787
|
+
database, or send a request that writes, under a step replay had already
|
|
788
|
+
run: the write ran, then was refused, and the extra row made the
|
|
789
|
+
`Then`'s right check fail. Such a write, and one the recording format
|
|
790
|
+
refuses, is refused before it runs. A check bound to a value of more
|
|
791
|
+
than about 32,000 characters that the steps spell out (a doc string's
|
|
792
|
+
JSON document) could not be recorded, the tool answering "Regular
|
|
793
|
+
expression too large": such a value is found as it stands.
|
|
794
|
+
- Outline rows: a literal date cell (`2026-11-20`) that a check showed in
|
|
795
|
+
the page's own format ("Friday, November 20, 2026", "20/11/2026"), an
|
|
796
|
+
amount cell an element showed with other decimals ("Pay $12.50" for
|
|
797
|
+
12.5, in a button's name or a text target), and a data row's value with
|
|
798
|
+
outer spaces (`" Oslo "` in a CSV) that the page showed trimmed were
|
|
799
|
+
found by no rule, so every other row checked row 1's value and a "not
|
|
800
|
+
shown" check passed whatever became of its row: the recording is bound
|
|
801
|
+
to its row. The page's ISO spelling of a cell such as `20.11.2026` no
|
|
802
|
+
longer becomes a date that moves with the calendar.
|
|
803
|
+
- Outline rows: a data table cell of three letters or more, or a
|
|
804
|
+
`{data:...}` value of four or more, was written into unrelated words a
|
|
805
|
+
check held ("Pro" in "Products", "Mark" in "Marketing"): after the cell
|
|
806
|
+
or the file changed, or on another row, a check looked for "Maxducts"
|
|
807
|
+
and a "not shown" check passed whatever the page showed. A value is
|
|
808
|
+
written only where it stands as a value, and an edit whose old value
|
|
809
|
+
stays in such a word records the step once more. A `{data:...}` value
|
|
810
|
+
shown inside angle brackets ("Ann <ann@shop.test>") was kept as recorded
|
|
811
|
+
and read as a placeholder, so the recording stayed fresh when the file
|
|
812
|
+
changed and checked the old address: it is written as its reference, and
|
|
813
|
+
a recording made before is recorded again.
|
|
814
|
+
- Outline rows: a data table value equal to the row's value ("alice"
|
|
815
|
+
seeded by a table, and the row's `<user>`) was written as the table's
|
|
816
|
+
cell in a later check ("Signed in as alice"), so every other row checked
|
|
817
|
+
alice's name: it is the row's `<user>`, or kept as recorded and bound to
|
|
818
|
+
its row where the check lists every row's value. A `{unique:<column>}`
|
|
819
|
+
the agent typed as this row's value was recorded as `{unique:alice}`,
|
|
820
|
+
and every other row was refused: it is recorded as the step writes it,
|
|
821
|
+
and each row types its own generated value. Such a recording is format
|
|
822
|
+
version 23, which older runners refuse: they typed the token as text. A
|
|
823
|
+
capture the agent named after the row's value (`alicebooking`) refused
|
|
824
|
+
every other row once a step naming the column read it: a capture's name
|
|
825
|
+
is no row data. An Outline value of about 33,000 characters or more (a
|
|
826
|
+
maximum-length text from a data file) crashed the recording beside an
|
|
827
|
+
attribute check or a response status: it records.
|
|
828
|
+
- Outline rows: a block beside `Examples: {data:roles}` that lacked one of
|
|
829
|
+
the data file's columns ran `<menu>` as written, on the data rows'
|
|
830
|
+
recording, so a row could pass on another row's data (two data files
|
|
831
|
+
missed it the same way when the placeholder was only in a doc string or
|
|
832
|
+
a table), and a placeholder with a space inside its brackets (`<role >`,
|
|
833
|
+
`< role >`) stayed as written on every row, so the rows replayed each
|
|
834
|
+
other's recording. Both are refused when the file is read, naming the
|
|
835
|
+
placeholder.
|
|
836
|
+
- Recording: a `Then` that read back what a `When` had typed was still
|
|
837
|
+
recorded when it checked the text of a textarea, a rich-text editor or a
|
|
838
|
+
select, named the field another way than the fill did (a selector, a
|
|
839
|
+
test id, a name pattern), or checked an Outline row's value the agent
|
|
840
|
+
had typed as its column: each passed whether or not the app saved
|
|
841
|
+
anything. The recording tool decides on the page which field a check
|
|
842
|
+
reads, and refuses these as it refused the value check. A check against
|
|
843
|
+
a value captured in the same `Then` was recorded when it named the
|
|
844
|
+
captured element another way (`[data-testid=total]` for the `h1`
|
|
845
|
+
captured) or a box holding it, comparing the element with itself: it is
|
|
846
|
+
refused, decided on the page too.
|
|
847
|
+
- Recording: a check found by the text it checks, beside a role (role
|
|
848
|
+
`cell`, text "Ada") or a selector scoped only by tags
|
|
849
|
+
(`table td:has-text('Ada')`), held on any cell showing that text; a
|
|
850
|
+
"starts with" or "ends with" check that let any text come between the
|
|
851
|
+
anchor and the value (`^.*ORD`, `2026.*$`) matched the value anywhere; a
|
|
852
|
+
URL check on what the app's base URL holds (`localhost:3000`, its base
|
|
853
|
+
path) held on every page; and a browser response check or network wait
|
|
854
|
+
passed with a JSON pattern matching any text, a status no step names
|
|
855
|
+
(the 500 the app happened to answer), or a class where the step names
|
|
856
|
+
the code (2xx for "201 Created"). Each is refused, the last as the API
|
|
857
|
+
medium's own checks refuse it.
|
|
858
|
+
- Recording: a password typed as `{env:...}` reached the recording agent
|
|
859
|
+
base64-encoded, in a Basic Authorization header that
|
|
860
|
+
`browser_network_request` shows or from page code that encoded the
|
|
861
|
+
field, and reports and traces kept such a value too, outside a trace's
|
|
862
|
+
request headers: a value inside base64 is masked to its token wherever
|
|
863
|
+
text is masked for display. A heal prompt's live tab line showed a page
|
|
864
|
+
URL holding a credential an API step had captured, while the line below
|
|
865
|
+
masked it: the credential is its `{captured:...}` token there and in the
|
|
866
|
+
completed steps. A first recording, not only a heal, filed the agent's
|
|
867
|
+
rewording of a `Then`, which `accept --all --with-feature-edit` then
|
|
868
|
+
wrote into the feature file: a `Then`'s wording is never filed as a
|
|
869
|
+
feature edit.
|
|
870
|
+
- Browser checks: text inside a box closed to no height or no width with
|
|
871
|
+
overflow hidden (a menu or an accordion closed with `max-height: 0`,
|
|
872
|
+
`height: 0` or `width: 0`) counted as shown in every browser, so "I see
|
|
873
|
+
Delete account" and "Delete account is not shown" both passed on one
|
|
874
|
+
page: it counts as not shown, and so do the fields, buttons and list
|
|
875
|
+
items such a box holds, in replay and exported tests alike, while what
|
|
876
|
+
is positioned against a box outside it still counts, as the browser
|
|
877
|
+
draws it. Text in a zero-sized inline `<svg>` is not shown either. A
|
|
878
|
+
target recorded inside an iframe was read as shown while the iframe was
|
|
879
|
+
hidden, zero-sized, off the page or in such a box, and its "not shown"
|
|
880
|
+
check passed too: it counts only while every iframe it was recorded in
|
|
881
|
+
is shown.
|
|
882
|
+
- Browser checks: a check that something is shown or holds a text, a
|
|
883
|
+
value, an amount, a style or an attribute, or that the URL holds a text,
|
|
884
|
+
was judged at its first look, before the request the `When` sent was
|
|
885
|
+
answered, so "the total is still $50.00", "I am still on the sign-up
|
|
886
|
+
page" and an optimistic "1 item" the server refused all passed: when the
|
|
887
|
+
action before it still has a request out, the check waits for the answer
|
|
888
|
+
(half the action timeout, 3 seconds at most: a slower answer is not
|
|
889
|
+
waited for) and passes only on a state that holds for half a second, as a count or a
|
|
890
|
+
"not shown" check does, in replay and exported tests alike. A check
|
|
891
|
+
whose action sent nothing is judged at once, as before.
|
|
892
|
+
- Browser checks: an attribute check in the absence form ("not disabled"
|
|
893
|
+
as `^(false)?$`) passed on an HTML boolean attribute that was there
|
|
894
|
+
(`disabled`, `checked`, `required`, `hidden`, `open` and the like),
|
|
895
|
+
whose empty value the pattern also takes, so "the Save button is not
|
|
896
|
+
disabled" held on a disabled button: such an attribute is set whenever
|
|
897
|
+
it is there, whatever its text. A style check on a shorthand (`outline`,
|
|
898
|
+
`background`) or a font list, recorded in Chromium, was red in WebKit or
|
|
899
|
+
Firefox on a page that looked the same: it is compared by the longhands
|
|
900
|
+
the value sets. Exported tests read the same.
|
|
901
|
+
- Replay: a page whose own script never yields (an endless loop after a
|
|
902
|
+
click) left every check, and the run, waiting for ever, with no red and
|
|
903
|
+
no report: the check fails at its time limit, saying the page did not
|
|
904
|
+
answer, its own script kept it busy, and the run goes on and writes its
|
|
905
|
+
report. A wait whose target has several ways to be found waited the
|
|
906
|
+
whole action timeout on each in turn before failing (four ways, four
|
|
907
|
+
timeouts): they are polled together under one deadline, the first that
|
|
908
|
+
finds an element deciding.
|
|
909
|
+
- API steps: a status a step named only to rule it out ("the API should
|
|
910
|
+
not answer 500", "a 4xx but not 404") counted as named: the agent was
|
|
911
|
+
told to expect it, and a recording that passed only when the API
|
|
912
|
+
answered 500 was kept. Such a status is no longer named, by a request's
|
|
913
|
+
expected status or by a `Then`'s status check. After copies that
|
|
914
|
+
answered otherwise, or a poll that ran out, the agent's next check was
|
|
915
|
+
verified against that unrecorded response, and the reply said the
|
|
916
|
+
request could not be sent: no check reads it now, the transcript marks
|
|
917
|
+
it as not recorded, and the reply says what it answered. A kept value
|
|
918
|
+
was quoted whole in the tool's reply, past the cap on a tool result (a
|
|
919
|
+
list of 6,000 products made a 418 KB reply): it is cut as a body is.
|
|
920
|
+
- API steps: a capture the scenario sent as its credential (an API key
|
|
921
|
+
kept as `key` and sent as `X-Api-Key`, a `Bearer {captured:result}`) was
|
|
922
|
+
a credential only when its name said so: the key reached the transcript,
|
|
923
|
+
the report, error text and the agent, and was sent to another host. A
|
|
924
|
+
capture sent in a credential header, or in a query parameter, form field
|
|
925
|
+
or body field a credential's name names, is a credential whatever its
|
|
926
|
+
name, from the moment it is extracted, in a browser scenario's API
|
|
927
|
+
steps too, where it became one only when a request first sent it, so a
|
|
928
|
+
check failing before then quoted it. An exported test keeps it as one
|
|
929
|
+
too: masked, sent only to the APIs under test, and with the trace left
|
|
930
|
+
out of the generated config.
|
|
931
|
+
- API steps: a schema check ignored a discriminator, so a valid response
|
|
932
|
+
of a `oneOf` picked by `discriminator` (the OpenAPI specification's own
|
|
933
|
+
Pet example) was always red, and such a request body could not be
|
|
934
|
+
recorded: the discriminator's property picks the schema, by its mapping
|
|
935
|
+
or the component's name, in replay and exported tests alike, and a value
|
|
936
|
+
neither names is red. A schema check was red on a response the contract
|
|
937
|
+
allows when its media type had no JSON schema (`"application/json": {}`,
|
|
938
|
+
or a `text/plain` response the contract gives a text schema): a media
|
|
939
|
+
type with no schema leaves the body free (a JSON one must still be
|
|
940
|
+
JSON), and a non-JSON type the contract gives is checked as text against
|
|
941
|
+
its schema. Behind `transport.api.proxy` the API was probed directly
|
|
942
|
+
before an agent session started, so an API whose host resolves only
|
|
943
|
+
through the proxy was never recorded ("did not answer (ENOTFOUND)"): the
|
|
944
|
+
probe goes to the proxy, as every request does, and names a proxy that
|
|
945
|
+
is down.
|
|
946
|
+
- Exported tests: an exported `@api` scenario sent its requests through
|
|
947
|
+
the test's browser context, which Playwright starts with the project's
|
|
948
|
+
`storageState`, so a request whose own token failed passed on the
|
|
949
|
+
signed-in session's cookie, where replay is red: each session of an
|
|
950
|
+
exported `@api` scenario sends through a request client of its own with
|
|
951
|
+
no cookies, as replay does, and a browser action in an `@api` scenario,
|
|
952
|
+
which replay fails, is exported as `test.fixme`. A failed exported
|
|
953
|
+
element check printed the resolved `{env:...}` value it waited for (a
|
|
954
|
+
text, a field's value, a URL, a download's text, or a target named by
|
|
955
|
+
one) in the terminal, the CI log and Playwright's reports: it names the
|
|
956
|
+
`{env:...}` token, as replay's errors do. `saffron eject` warned only of
|
|
957
|
+
a Playwright trace, and said turning it off was enough, while
|
|
958
|
+
Playwright's HTML report keeps every typed `{env:...}` value as a step's
|
|
959
|
+
title: it warns of an HTML or blob reporter in the project's own
|
|
960
|
+
Playwright config too, naming `playwright-report` and `blob-report`
|
|
961
|
+
beside `test-results`.
|
|
962
|
+
- Runs: a heal that recorded the same actions as the committed recording,
|
|
963
|
+
differing only in the page fingerprint a browser recording stamps, was
|
|
964
|
+
filed as a proposal, so a failing step turned yellow: it is red, as any
|
|
965
|
+
heal that changed nothing is. A scenario with no steps of its own (its
|
|
966
|
+
lines read as its description) ran its Background alone and could pass:
|
|
967
|
+
it is red, as a scenario with no steps is. With the app down, a pending
|
|
968
|
+
proposal's replay was blamed on the recording (for another Examples row,
|
|
969
|
+
with advice to reject it and record again) or, under `--no-agent`, on
|
|
970
|
+
the recorder: it is blocked by the environment, as a committed
|
|
971
|
+
recording's replay is. A scenario that needed recording in a browser
|
|
972
|
+
that only replays (Firefox, WebKit) was counted as blocked by the
|
|
973
|
+
agent's provider: it is blocked by the recorder. With no agent and
|
|
974
|
+
several workers, checking a recording again against a changed API
|
|
975
|
+
contract ran the `reset` commands while other workers replayed: it runs
|
|
976
|
+
none, as an ordinary replay.
|
|
977
|
+
- Review: a held note whose own words held a value the scenario names with
|
|
978
|
+
`{env:...}` (`stable` in `unstable recording:`, `base` in
|
|
979
|
+
`database write added`) was masked there too, and `accept --all` then
|
|
980
|
+
promoted the proposal: a note's prefix is never masked. A heal that
|
|
981
|
+
changed or dropped the status a request expects (201 to 409, or to any
|
|
982
|
+
2xx) kept that request's requirement and was not held for review: the
|
|
983
|
+
expected status is part of it. An import that stopped part way (a write
|
|
984
|
+
that failed, Ctrl-C) left the proposals it had already written out of
|
|
985
|
+
the import ledger, and `accept --all` promoted them as if filed here:
|
|
986
|
+
each is in the ledger before it is written. `saffron run --rerecord`
|
|
987
|
+
lost track of the recording it replaced when that one was kept under its
|
|
988
|
+
0.9.9 file name (a scenario name with no ASCII letters, or a long name
|
|
989
|
+
saved whole), so a looser `Then` recorded again was not held: it is
|
|
990
|
+
found. A `--rerecord` run that recorded nothing for a scenario deleted
|
|
991
|
+
the recording it had set aside when the scenario's pending proposal was
|
|
992
|
+
rejected or replaced during the run: it is put back.
|
|
993
|
+
- Configuration: a count or limit written in quotes
|
|
994
|
+
(`"stabilityProofs": "2"`) was read as a number by `saffron run` and as
|
|
995
|
+
text by the MCP `run_plan`, which planned 1121 executions: every command
|
|
996
|
+
refuses it as the config loads, naming the key. A bad value in
|
|
997
|
+
`saffron.config.json` was refused as a flag the user never passed
|
|
998
|
+
(`invalid --trace "yes"`): the message names the key
|
|
999
|
+
(`invalid trace "yes" in saffron.config.json`), and the flag only when
|
|
1000
|
+
one was given. A config saved with a byte order mark (Windows Notepad,
|
|
1001
|
+
PowerShell 5.1) was refused as not valid JSON, and `saffron init` filled
|
|
1002
|
+
`features/` instead of the configured folder: the mark is read as no
|
|
1003
|
+
part of the JSON, as in a data file. `saffron mcp` did not start while
|
|
1004
|
+
`saffron.config.json` was broken, so the agent fixing it had no tools:
|
|
1005
|
+
the server starts, and each tool answers on the defaults with
|
|
1006
|
+
`configProblem` saying what is wrong.
|
|
1007
|
+
- `saffron init` replaced a `saffron` MCP server you had set up yourself
|
|
1008
|
+
(a local CLI, an `env`) with its `npx` entry and said only "updated": it
|
|
1009
|
+
keeps one that runs another command and says so, keeps any field you
|
|
1010
|
+
added to its own entry, and brings an earlier init's `npx saffron mcp`
|
|
1011
|
+
up to date. `saffron report --merge` of a named run's shards wrote
|
|
1012
|
+
`latest.json` and `history.jsonl` without the lock a run takes, so a run
|
|
1013
|
+
writing at the same time could lose its rows, or the merge its own: both
|
|
1014
|
+
write under one lock. `saffron status`, the editors' panels and the MCP
|
|
1015
|
+
`project_status` crashed on a last report whose row held steps or
|
|
1016
|
+
evidence that were not a list, or an error that was not text, and on one
|
|
1017
|
+
with no totals: such a row is read without those fields, and such a
|
|
1018
|
+
report as none.
|
|
1019
|
+
- Parser: a mistyped header (`scenario:`, `Background :`, or
|
|
1020
|
+
`Scenario Name` and `Background` with no colon) over a single step, or
|
|
1021
|
+
over `*` steps, was read as description and silently did not run, and a
|
|
1022
|
+
file of one such scenario ran nothing: it is refused, naming the header.
|
|
1023
|
+
A `*` line counts as a step only right under a header misspelt with its
|
|
1024
|
+
colon, and one step under a header word with no colon only when it is
|
|
1025
|
+
indented deeper, so bullets and prose in a description stay description.
|
|
1026
|
+
A StepSet call written after a step keyword (`When StepSet Login`,
|
|
1027
|
+
`* StepSet Login`) ran as an ordinary step the agent made sense of, and
|
|
1028
|
+
the set never ran: it is refused, saying a call is a line of its own. A
|
|
1029
|
+
StepSet at the top of a `.feature` file was told to fix its header's
|
|
1030
|
+
spelling, or with one step under it ran nothing: it is told to rename
|
|
1031
|
+
the file to `.saffron`. A refusal of a line read as description names
|
|
1032
|
+
that line, where it named its block's header (the language server marks
|
|
1033
|
+
it there too), and Gherkin's parse errors quote the line as written, at
|
|
1034
|
+
file:line:column, where they quoted it as Saffron rewrites it
|
|
1035
|
+
(`* StepSet Login`) and lost the file:line form in a file that defines a
|
|
1036
|
+
StepSet.
|
|
1037
|
+
- The language server, which JetBrains reads: one StepSet problem anywhere
|
|
1038
|
+
(a nested call, a name defined twice, a set with Examples, a set file
|
|
1039
|
+
that does not parse) left it no step sets, so every call in every file
|
|
1040
|
+
was marked "not defined anywhere", completion offered nothing, and the
|
|
1041
|
+
problem the runner stops on was marked nowhere: each problem is marked
|
|
1042
|
+
on its own line, and every other set is kept. A `.saffron` description
|
|
1043
|
+
line starting with "StepSet " is no longer marked as a call. Two
|
|
1044
|
+
scenarios of one name in a file (one per Rule), or names that slug
|
|
1045
|
+
alike, which every run of the file refuses as sharing a recording, are
|
|
1046
|
+
marked on the later one. A step with a long run of spaces inside it took
|
|
1047
|
+
more than a second per keystroke: the pattern sessions are read with is
|
|
1048
|
+
linear.
|
|
1049
|
+
- Exported tests: a check of a target inside an iframe passed while the
|
|
1050
|
+
iframe was hidden, zero-sized or inside a box clipped to nothing, where
|
|
1051
|
+
replay fails it: the exported test reads such a target as not shown. On
|
|
1052
|
+
a page that holds a request open (a long-poll, a notifications stream)
|
|
1053
|
+
an exported test waited up to 3 seconds before each check and failed a
|
|
1054
|
+
message shown briefly that replay passed: every action now starts its
|
|
1055
|
+
own window, as in replay.
|
|
1056
|
+
- A DynamoDB "not in the database" check filtering on a top-level
|
|
1057
|
+
attribute whose name holds a dot (`meta.tier`) was refused as naming an
|
|
1058
|
+
attribute no item has: it is read as the attribute it names.
|
|
1059
|
+
- Recordings and files that may turn red or stop a run with this release:
|
|
1060
|
+
a feature file with a mistyped scenario header (fix the header), a
|
|
1061
|
+
database check that counts rows under a one-row subquery (re-record,
|
|
1062
|
+
`saffron run --rerecord`), a "not shown" check on a whole number shown
|
|
1063
|
+
with zero decimals, an amount check on a value shown with three or more
|
|
1064
|
+
decimals that was a little off, a config with `actionTimeoutMs: 0`, an
|
|
1065
|
+
Outline that uses a `<column>` one of its Examples blocks lacks (add the
|
|
1066
|
+
column), a scenario with no name (name it), a Set-Cookie pattern written
|
|
1067
|
+
with a second branch, a "differs" check between a price and a text
|
|
1068
|
+
that is no amount (capture the amount alone), exported tests, once
|
|
1069
|
+
ejected again, that passed on text across a block's edge or a download's
|
|
1070
|
+
lines, a "not in the database" check whose bind is of another type than
|
|
1071
|
+
the database keeps the value as (give the bind its type), a `LIKE` bind
|
|
1072
|
+
with a backslash in such a check, a `$0` placeholder, a SQL Server alias
|
|
1073
|
+
that is a statement's word, an Oracle check that calls a `UTL_` or
|
|
1074
|
+
`DBMS_` package, a count of rows under a latest-row anti-join, a
|
|
1075
|
+
MySQL `UPDATE` of a `FLOAT` under cleanup (set `db.cleanup` to
|
|
1076
|
+
`"inserts"`), a config whose `baseURL` is not an `http://` or
|
|
1077
|
+
`https://` URL, a schema check on a response served as a type that is
|
|
1078
|
+
not JSON, a mistyped header over a single step or `*` steps, or a
|
|
1079
|
+
description line starting with Scenario, Example or Background over a
|
|
1080
|
+
step indented deeper (fix the header, or reword or unindent the line), a
|
|
1081
|
+
StepSet call after a step keyword (drop the keyword), a placeholder with
|
|
1082
|
+
a space inside its brackets (remove the space), an Examples block beside
|
|
1083
|
+
a `{data:...}` one that lacks one of its columns (add the column), a
|
|
1084
|
+
scenario with no steps of its own, only the Background's (give it steps,
|
|
1085
|
+
or remove it), a count or limit in quotes in `saffron.config.json`
|
|
1086
|
+
(`"stabilityProofs": "2"`: write the number), a recording whose `Then`
|
|
1087
|
+
writes to the database or whose `Given` or `When` checks it (re-record),
|
|
1088
|
+
a SQL check that a column is not in a row (re-record, and it is written
|
|
1089
|
+
`IS NULL`), a "not in the database" check on a field no document of the
|
|
1090
|
+
collection holds (fix a misspelt name, or let a document hold the field
|
|
1091
|
+
first), one that compares a JSON value with a bind read as JSON, a MySQL
|
|
1092
|
+
JSON boolean with 1 or 0, or a binary column with `0x` hex, or that
|
|
1093
|
+
binds JSON the SQL makes itself (compare as its message says), a count
|
|
1094
|
+
or a "no second row" check under a `WHERE` that keeps one row of each
|
|
1095
|
+
group (re-record), a check of a state shown only while the `When`'s
|
|
1096
|
+
request is out, such as a spinner (re-record, so it is watched for from
|
|
1097
|
+
the `When`), a check of text in a box closed to no height or width, or
|
|
1098
|
+
in an iframe the page hides (open it in a `When` first), an absence
|
|
1099
|
+
check on an HTML boolean attribute that is there with a text such as
|
|
1100
|
+
`false` (the browser reads it as set: fix the page), an Outline whose
|
|
1101
|
+
check shows a row's date, amount or spaced value in the page's own form
|
|
1102
|
+
(give the shown value an Examples column of its own), and exported
|
|
1103
|
+
`@api` tests, once ejected again, that passed on the signed-in session's
|
|
1104
|
+
cookie (give the request credentials of its own). From the last review:
|
|
1105
|
+
an Outline whose not-shown check holds the recorded row's day by its
|
|
1106
|
+
month's name without the year (the other rows are refused, and a run with
|
|
1107
|
+
no agent calls an older such recording stale: name the date column in the
|
|
1108
|
+
step, or re-record); a database "differs" check on text that spells a
|
|
1109
|
+
number (`"type": "string"` compares the text); a database "at most" check
|
|
1110
|
+
with an empty or null value (give the step a value, or count empty
|
|
1111
|
+
columns by `length(note) = $1` with 0; on Oracle an empty text is NULL,
|
|
1112
|
+
so write `IS NULL` beside a column compared with a bind); database steps with a driver
|
|
1113
|
+
of another major version than the tests run (the refusal names the one
|
|
1114
|
+
to install); a step whose select, fill, hover, drag, upload, check or
|
|
1115
|
+
key press passed only on its second try, after its handler kept the page
|
|
1116
|
+
busy past the action timeout (it is no longer sent twice: raise
|
|
1117
|
+
`actionTimeoutMs`); a click, a key press, a check, a drag or a hover that closes
|
|
1118
|
+
its own tab and is rejected by Playwright for it, where a second try used to
|
|
1119
|
+
land on the tab behind (the step is red at once); a recording that
|
|
1120
|
+
relied on Playwright 1.64 acting inside an `inert` part of the page, on
|
|
1121
|
+
an accessible name that holds text the browser does not draw, or on a
|
|
1122
|
+
cookie whose `Path` is only a prefix of the request's (re-record); a
|
|
1123
|
+
database "differs" check typed as a number or a boolean on a value the
|
|
1124
|
+
row holds as another type (give it the type the row shows, or none); a
|
|
1125
|
+
MySQL or MariaDB "not in the database" or "at most" check that binds an
|
|
1126
|
+
unquoted `true` on a `BOOLEAN` or number column holding false or 0 (bind
|
|
1127
|
+
1 or 0); an
|
|
1128
|
+
Outline whose not-shown check holds a word that is also a month's name
|
|
1129
|
+
("may", "march") beside a number that is a row's day (the other rows are
|
|
1130
|
+
refused: name the date column in the step); and exported tests, once
|
|
1131
|
+
ejected again, whose nine or more checks straight after a page load that
|
|
1132
|
+
keeps a request open now each wait about 3 seconds (raise the test's
|
|
1133
|
+
`timeout` in `playwright.config.ts`).
|
|
1134
|
+
|
|
1135
|
+
Editors (VS Code extension 0.8.3, JetBrains plugin 0.8.3)
|
|
1136
|
+
|
|
1137
|
+
- VS Code: the same headers and steps Gherkin reads as description are
|
|
1138
|
+
marked as errors; a StepSet the file itself defines is no longer called
|
|
1139
|
+
undefined, and a description line starting with "StepSet " is text; a
|
|
1140
|
+
`Background:` line under a scenario's header no longer hides its
|
|
1141
|
+
Examples errors; a `.feature` file in a folder with no
|
|
1142
|
+
`saffron.config.json` gets no Saffron errors; a row of a report written
|
|
1143
|
+
on Windows opens its scenario; with no folder open the panel says so
|
|
1144
|
+
instead of loading for ever; `{data:...}` hints read a data file once
|
|
1145
|
+
until it changes. More of what the runner's parser refuses is marked
|
|
1146
|
+
where it is written: a scenario with no name, an outline placeholder
|
|
1147
|
+
no Examples row fills (a column only another Examples block has, a
|
|
1148
|
+
near miss of a column, a column named twice, placeholders with no
|
|
1149
|
+
Examples, a column a data file lacks), a `{unique:...}` that is no
|
|
1150
|
+
token, a StepSet line anywhere in a `.feature` file, and a fixture or
|
|
1151
|
+
step set the runner cannot run.
|
|
1152
|
+
- JetBrains: the CI Runs results page runs none of the scripts a run's
|
|
1153
|
+
bundle carries; a row of a report written on Windows opens its scenario
|
|
1154
|
+
at its line; a feature file that is not valid UTF-8 is listed with its
|
|
1155
|
+
scenarios; Run Selected on one file names its configuration by the
|
|
1156
|
+
file's path; a test folder outside the project is watched; a selected
|
|
1157
|
+
scenario row in CI Runs stays selected through a reload, told apart
|
|
1158
|
+
from another feature's of the same name. A folder of feature files
|
|
1159
|
+
added to the test folder is listed without a Refresh, and on Windows a
|
|
1160
|
+
test folder on another drive is listed at all; a status reload keeps
|
|
1161
|
+
the proposal being read and the Last Run row selected; while the status
|
|
1162
|
+
loads, the tabs say so instead of claiming nothing is pending; the
|
|
1163
|
+
Accept All tooltips name a database write a new recording added.
|
|
1164
|
+
|
|
1165
|
+
## 1.4.0: Database steps
|
|
1166
|
+
|
|
1167
|
+
Released 2026-11-10.
|
|
1168
|
+
|
|
1169
|
+
Runner
|
|
1170
|
+
|
|
1171
|
+
- Database steps (beta): a step like `Then the customer should
|
|
1172
|
+
be saved in the database` works with no query written by hand. While
|
|
1173
|
+
recording, the agent reads the schema and records one parameterized
|
|
1174
|
+
`SELECT`, its values as binds; every later run replays it at zero
|
|
1175
|
+
tokens. A check must be bound to a value the scenario wrote; a value that
|
|
1176
|
+
is new on each run (`{unique:...}`, an id an earlier step kept) keeps it
|
|
1177
|
+
from passing on rows an earlier run left. With `"writes": true` and the
|
|
1178
|
+
database's host or name in `allow`, a `Given` or `When` may seed data with
|
|
1179
|
+
one `INSERT` or `UPDATE ... WHERE`. When the scenario ends, green or
|
|
1180
|
+
red, `db.cleanup` undoes them: `"all"` (default) deletes the rows it
|
|
1181
|
+
inserted and puts the rows it updated back as they were, `"inserts"`
|
|
1182
|
+
deletes inserts only, `"none"` leaves everything. `DELETE`, `TRUNCATE`,
|
|
1183
|
+
DDL and several statements are refused by a SQL parser. Set `db` in
|
|
1184
|
+
`saffron.config.json` and `npm install node-sql-parser` with the
|
|
1185
|
+
engine's driver: neither comes with saffron-ai, so a project without
|
|
1186
|
+
database steps does not carry the parser's 88 MB. Recordings with
|
|
1187
|
+
database steps are cache version 20.
|
|
1188
|
+
- Database steps on PostgreSQL, MySQL and SQL Server, named by db.url's
|
|
1189
|
+
scheme (`postgres://`, `mysql://`, `mssql://`), with the driver `pg`,
|
|
1190
|
+
`mysql2` or `mssql`. Each engine takes the same steps, rules and `$1` binds; MySQL and SQL Server writes
|
|
1191
|
+
are read back by their primary key, one row per `INSERT` when the key
|
|
1192
|
+
is auto-numbered. A recording made on one engine is refused on another.
|
|
1193
|
+
A write to more than one table (MySQL's `UPDATE a JOIN b`) is refused,
|
|
1194
|
+
and a write whose rows cannot be read back by their key is rolled back,
|
|
1195
|
+
so cleanup never misses a row or touches one Saffron did not write. A
|
|
1196
|
+
SQL Server check that meets a row the app is still writing reads again
|
|
1197
|
+
instead of failing. A table hint (`WITH (NOLOCK)`, `READPAST`) is refused,
|
|
1198
|
+
so a check never reads rows the app did not commit or skips rows it
|
|
1199
|
+
holds. A write goes only to the database `db.url` names (MySQL's
|
|
1200
|
+
`otherdb.t` and SQL Server's `db.schema.t` are refused), an `UPDATE` with
|
|
1201
|
+
`ORDER BY` or `LIMIT` is refused, and a SQL Server `UPDATE` that changes a
|
|
1202
|
+
time finer than milliseconds, or with an offset, is refused while cleanup
|
|
1203
|
+
would put it back. MySQL's `INSERT IGNORE` is refused, as it skips rows
|
|
1204
|
+
without saying so, and a check whose rows the app holds locked for the
|
|
1205
|
+
whole `db.timeout` fails saying so.
|
|
1206
|
+
- A database check reads again within `db.timeout` (default 3000 ms): a
|
|
1207
|
+
presence check passes as soon as its rows are there, and a "not in the
|
|
1208
|
+
database" check waits for a row a worker deletes late, then must hold
|
|
1209
|
+
to the end of the window, so a row a worker writes late fails it.
|
|
1210
|
+
- On PostgreSQL and MySQL every statement a database step runs goes to
|
|
1211
|
+
the database as exactly one statement, so a query the SQL check reads
|
|
1212
|
+
as one SELECT can never run a second statement (a quote read differently
|
|
1213
|
+
by the two could hide a COMMIT and a DELETE in a look at the data). SQL
|
|
1214
|
+
Server runs a batch, so there the parser's one-statement rule and the
|
|
1215
|
+
refused backslash before a quote hold it.
|
|
1216
|
+
- Cleanup puts a json or jsonb column back as JSON, an array or a string
|
|
1217
|
+
value included. Placeholders numbered with a gap ($1 and $3, no $2) are
|
|
1218
|
+
refused while recording.
|
|
1219
|
+
- Only a scenario whose recording has database steps, or whose steps name
|
|
1220
|
+
the database before it is recorded, needs the database to answer before
|
|
1221
|
+
recording starts.
|
|
1222
|
+
- A database check that the database refused (a permission, a timeout)
|
|
1223
|
+
says so, apart from one that found other rows; neither starts an agent.
|
|
1224
|
+
- A database check whose stored query fails because the schema changed
|
|
1225
|
+
is repaired by the agent, the one exception to a `Then` never being
|
|
1226
|
+
healed: only its table, columns and joins may change, its binds and
|
|
1227
|
+
expected rows are enforced unchanged, it must return as many rows as
|
|
1228
|
+
the recording did (a join that fans out is refused), and the repair must
|
|
1229
|
+
pass a proof replay. The step is reported healed and a proposal names the old and
|
|
1230
|
+
new SQL. A query that returns the wrong rows stays red, with no agent.
|
|
1231
|
+
- The run report shows the rows each database check found under its
|
|
1232
|
+
step, masked, the first five.
|
|
1233
|
+
- A recorded check must tie its rows to the scenario's value: a column
|
|
1234
|
+
compared with a bind by `=`, `IN` or `LIKE`, on each side of an `OR`,
|
|
1235
|
+
and a subquery on the tying side ties its own rows the same way.
|
|
1236
|
+
`UNION`, `INTERSECT` and `EXCEPT`, `email <> $1`, `NOT (...)`, a
|
|
1237
|
+
comparison inside `COALESCE` or `CASE`, a range alone
|
|
1238
|
+
(`created_at > $1`) and a bind in `LIMIT` or `OFFSET` are refused, as
|
|
1239
|
+
each could hold on rows the scenario never wrote. The same holds for an
|
|
1240
|
+
`UPDATE`'s `WHERE`. A JSON field can tie a check
|
|
1241
|
+
(`data->>'orderRef' = $1`, `data @> CAST($1 AS jsonb)`). A null bind,
|
|
1242
|
+
or an empty one in a "not in the database" check, is refused while
|
|
1243
|
+
recording and at replay: `= NULL` matches no row.
|
|
1244
|
+
- A repaired database check, and a heal that adds or changes a database
|
|
1245
|
+
write, are held for review: `accept --all` leaves them out (`DATABASE
|
|
1246
|
+
CHECK REPAIRED`, `DATABASE WRITE ADDED`).
|
|
1247
|
+
- A database that is down, restarts mid-step or does not answer within 10
|
|
1248
|
+
seconds is the environment: no agent session starts, and the message
|
|
1249
|
+
says the database did not answer, not the app. Before recording, an
|
|
1250
|
+
unset `db.url` variable, a connection string that is not a URL, a wrong
|
|
1251
|
+
password, a missing database and a missing driver (`pg`, `mysql2` or
|
|
1252
|
+
`mssql`) or `node-sql-parser` are said at once, without starting an
|
|
1253
|
+
agent.
|
|
1254
|
+
- `saffron status --json` and the `project_status` MCP tool say whether
|
|
1255
|
+
database steps are configured (`database`: the engine, whether writes
|
|
1256
|
+
are on, the cleanup mode), never the connection string; they and the
|
|
1257
|
+
editors' config views mask a password in a `db.url` written in keyword
|
|
1258
|
+
form (`host=... password=...`) or in its query (`?password=`). The
|
|
1259
|
+
bundled skill, `saffron init`'s AGENTS.md block and the guides cover
|
|
1260
|
+
database steps and the one Then whose query may be repaired.
|
|
1261
|
+
- `run --plan` and the MCP `run_plan` tool list a recorded database write
|
|
1262
|
+
as a side effect when `db.cleanup` keeps it (`"inserts"` keeps an
|
|
1263
|
+
`UPDATE`, `"none"` every write).
|
|
1264
|
+
- An `@api` scenario whose recording sends no request (database steps
|
|
1265
|
+
alone) replays without the API answering.
|
|
1266
|
+
- The language server (JetBrains completion and hover) says a step whose
|
|
1267
|
+
recordings all read or write the database replays at zero tokens only in
|
|
1268
|
+
the scenario that recorded it, per `As "..."` session, as VS Code does,
|
|
1269
|
+
and its similar-wording hint no longer suggests such a step.
|
|
1270
|
+
- Found in the release review, before release:
|
|
1271
|
+
- A write inside a `WITH` passed when both statements were `UPDATE`s,
|
|
1272
|
+
and its table was never put back. Any `WITH` in a write is refused,
|
|
1273
|
+
and so is a `WHERE` that reads no column (`WHERE true`).
|
|
1274
|
+
- A check could echo its bind back (`SELECT $1 AS email`), or hold on
|
|
1275
|
+
any row (`OR true`, `OR email IS NOT NULL`, `(email = $1) IS NOT NULL`,
|
|
1276
|
+
`$1 = $1`), and a schema repair could add
|
|
1277
|
+
`LIMIT 1`, `DISTINCT` or `max()` to hide a join that fans out. Both
|
|
1278
|
+
are refused, and so are values written in as `E'...'`, `$$...$$` or
|
|
1279
|
+
`DATE '...'`.
|
|
1280
|
+
- Under `"assertionPolicy": "adaptable-mid"`, a heal rewrote a database
|
|
1281
|
+
check in the middle of a scenario, its binds and expected rows too.
|
|
1282
|
+
A database check is never healed, under every policy.
|
|
1283
|
+
- A database check recorded by another scenario was reused with that
|
|
1284
|
+
scenario's value, so it could pass on rows an earlier run left. A
|
|
1285
|
+
database step is seeded only from the scenario's own recording.
|
|
1286
|
+
- A `db.url` written as `host=... password=...` was shown whole in the
|
|
1287
|
+
error, password included. The message no longer shows the value, and
|
|
1288
|
+
the rows an error shows are masked before they are cut.
|
|
1289
|
+
- A database connection dropped mid-step crashed the run, a host that
|
|
1290
|
+
does not answer now fails within 10 seconds, and a step still running
|
|
1291
|
+
after Ctrl-C opens no new connection.
|
|
1292
|
+
- Ctrl-C during a database write left its row behind. Closing now
|
|
1293
|
+
waits for the write and undoes it.
|
|
1294
|
+
- A check on an aggregate's one row passed whatever the table held:
|
|
1295
|
+
`count(*)` with no `GROUP BY`, an aggregate the parser reads as a
|
|
1296
|
+
plain function (`json_agg`, `JSON_ARRAYAGG`, `bool_or`), `GROUP BY
|
|
1297
|
+
ROLLUP`, and a query whose only `FROM` or `WITH` source is an
|
|
1298
|
+
aggregate. A count, exists or item check on that row is refused while
|
|
1299
|
+
recording and at replay; only a value checked on it is taken.
|
|
1300
|
+
- A count above 0, or a check that there is no second row, could not see
|
|
1301
|
+
a second row under `LIMIT`, `TOP`, `DISTINCT` or a `GROUP BY` with no
|
|
1302
|
+
aggregate, and `OFFSET` (SQL Server's `OFFSET ... FETCH` too) hid rows
|
|
1303
|
+
from a count and let a "no rows" check pass: each is refused. On MySQL
|
|
1304
|
+
and SQL Server a check never saw `DISTINCT` (the parser writes it
|
|
1305
|
+
differently there), so such checks passed on a duplicate row and a
|
|
1306
|
+
repair could add it; it is now read on every engine.
|
|
1307
|
+
- A check that expects no rows, or that a field must not be there, is no
|
|
1308
|
+
longer repaired: a repaired query finding none proved nothing, and a
|
|
1309
|
+
repair could stop selecting that field.
|
|
1310
|
+
- A field that must not be in a row (`/0/data/discount`) passed when the
|
|
1311
|
+
app saved no row at all; it now needs the row to be there.
|
|
1312
|
+
- A captured id of one or two characters was recorded as itself in a
|
|
1313
|
+
database bind, so replay checked the recording run's row; it must be
|
|
1314
|
+
written as `{captured:name}`, and one written as itself beside another
|
|
1315
|
+
bind is refused.
|
|
1316
|
+
- On MySQL, `--` with no space after it let a look at the data run what
|
|
1317
|
+
Saffron read as a comment; it is refused. On SQL Server, the agent's
|
|
1318
|
+
`UPDATE` was read as PostgreSQL when checking that its `WHERE` names
|
|
1319
|
+
the scenario's rows, and a write to a table whose key the driver reads
|
|
1320
|
+
inexactly (a decimal past 15 digits) is refused while cleanup is on.
|
|
1321
|
+
- A page fingerprint left by a browser tool that failed went onto the
|
|
1322
|
+
next database or HTTP action, and that recording was refused on every
|
|
1323
|
+
load after.
|
|
1324
|
+
- A MySQL write could read back a row that was there before (`7`
|
|
1325
|
+
finding `'07'`) for one its key missed, and cleanup deleted it: each
|
|
1326
|
+
key now reads back exactly its own row, or the write is rolled back.
|
|
1327
|
+
A JSON column is put back as the text it held, never read through a
|
|
1328
|
+
JS number.
|
|
1329
|
+
- A secret a driver quotes cut short (MySQL's duplicate entry at 64
|
|
1330
|
+
characters, SQL Server's truncated value), one holding a parenthesis
|
|
1331
|
+
or cut to 4 characters too, is masked, and a secret in a row no longer
|
|
1332
|
+
breaks the rows the report shows. A lock held for the whole window
|
|
1333
|
+
says it timed out, not that it found other rows.
|
|
1334
|
+
- A database check was seeded from the scenario's own recording after
|
|
1335
|
+
an earlier step changed the value it binds; it is recorded again. A
|
|
1336
|
+
stale recording whose edited steps name the database now asks the
|
|
1337
|
+
database to answer before recording.
|
|
1338
|
+
- A SQL Server `BIGINT` identity past 2^53 is read exactly, so cleanup
|
|
1339
|
+
deletes the scenario's row, never the app's.
|
|
1340
|
+
- A check or an agent `UPDATE` with an `OR` whose other side is a fixed
|
|
1341
|
+
value (`email = $1 OR role = 'admin'`) passed on rows earlier runs
|
|
1342
|
+
left: each side of an `OR`, and every value of an `IN`, must hold a
|
|
1343
|
+
value the scenario wrote. A `LIKE` value of only `%` or `_` (from an
|
|
1344
|
+
Examples cell too) is refused unless another condition ties the rows,
|
|
1345
|
+
and so are `LIMIT 0`, `TOP 0` and `FETCH ... 0`.
|
|
1346
|
+
- A quoted, qualified column (`` `o`.`total` ``, `"o"."total"`) was
|
|
1347
|
+
refused as a value written into the SQL; it is read as a name.
|
|
1348
|
+
- Stability proofs and an Outline's row proofs failed on rows the first
|
|
1349
|
+
proof seeded: its database writes are now undone before they run.
|
|
1350
|
+
- The MCP `search_steps` tool showed the agent a database step's
|
|
1351
|
+
session-prefixed text (`As "name" ...`), which would move that step
|
|
1352
|
+
into the named session; it no longer does.
|
|
1353
|
+
- An Outline check that holds a row's value as the page sets it in type
|
|
1354
|
+
(a curly apostrophe or quotes, spaces collapsed or no-break, an
|
|
1355
|
+
ellipsis) let another row pass on it; that row now records again or is
|
|
1356
|
+
refused.
|
|
1357
|
+
- A key under `api` or `api.services.<name>` that Saffron does not read
|
|
1358
|
+
(`"transcripts"`, `"contracts"`) is refused with the keys it takes,
|
|
1359
|
+
instead of leaving that setting at its default.
|
|
1360
|
+
- A check that something is shown (text, visible, a field's value) no
|
|
1361
|
+
longer passes on content inside a frame the page hides (`visibility:
|
|
1362
|
+
hidden`, zero-sized, placed off the page), or inside a frame in one.
|
|
1363
|
+
- A base URL's password, which Chromium keeps in the page's URL, is
|
|
1364
|
+
masked in what the recording agent reads: snapshots, tool errors,
|
|
1365
|
+
Saffron tool replies and the heal prompt's live page and failure.
|
|
1366
|
+
Navigating to the masked URL still signs in.
|
|
1367
|
+
- A remote browser's endpoint (`transport.browser`) no longer prints its
|
|
1368
|
+
password or `?token=` key when the connection fails.
|
|
1369
|
+
- `saffron status --json`, the `project_status` and `api_contract` MCP
|
|
1370
|
+
tools mask a password in a base URL and a key in a URL's query or
|
|
1371
|
+
fragment (`?token=`, `?api_key=`, `?key=`, `#access_token=`); `api_contract` showed the password
|
|
1372
|
+
to the agent. The MCP `verify` tool masks them in a failed run's last
|
|
1373
|
+
lines, and `saffron api pin` in what it prints.
|
|
1374
|
+
- `saffron api pin` says why a URL could not be fetched (connection
|
|
1375
|
+
refused, host not found) instead of only "fetch failed".
|
|
1376
|
+
- `saffron run --json` without `--plan` exits 2 instead of starting a run
|
|
1377
|
+
that prints no JSON.
|
|
1378
|
+
- `saffron eject --out <dir>` warns when the project's Playwright config
|
|
1379
|
+
looks for tests in another folder, so `npx playwright test` would not
|
|
1380
|
+
run the new specs.
|
|
1381
|
+
- `saffron init` registers the MCP server as `npx -y -p saffron-ai saffron
|
|
1382
|
+
mcp`: a bare `npx saffron` fetched npm's unrelated `saffron` package in
|
|
1383
|
+
a project without saffron-ai installed.
|
|
1384
|
+
- A `saffron.config.json` that is not valid JSON is named in the error.
|
|
1385
|
+
- `saffron status --json` and the `project_status` MCP tool name, in each
|
|
1386
|
+
`lastRun.attention[]` entry, why a red was not run (`blocked`:
|
|
1387
|
+
`recorder`, `environment`, `provider` or `budget`), so editors list it
|
|
1388
|
+
as not run instead of failed.
|
|
1389
|
+
- `--project-root` naming a folder that does not exist stops with exit 2,
|
|
1390
|
+
instead of reading it as an empty project, and `saffron run` names the
|
|
1391
|
+
configured features folder when it does not exist.
|
|
1392
|
+
- A run notes when the selected data environment (`--env`, `SAFFRON_ENV`
|
|
1393
|
+
or `env`) matches no data file, so a misspelled name no longer uses the
|
|
1394
|
+
base data in silence.
|
|
1395
|
+
- `Examples: {data:...}` refuses a CSV file that names a column twice, as
|
|
1396
|
+
an Examples table does; one of them was read for both. A data JSON file
|
|
1397
|
+
saved with a byte order mark is read.
|
|
1398
|
+
- The recording agent can no longer send an `{env:...}` value to another
|
|
1399
|
+
site by writing the site as a reference (`{captured:link}?p={env:...}`,
|
|
1400
|
+
a row's `<column>`, a `{data:...}` value): the site is checked as the
|
|
1401
|
+
browser will open it.
|
|
1402
|
+
- A capture, and a check that a value changed, wait while the element
|
|
1403
|
+
they were recorded on is still empty, instead of reading the same test
|
|
1404
|
+
id in an iframe or a fallback that happens to show text: "the price
|
|
1405
|
+
changed" passed on an embedded widget's copy.
|
|
1406
|
+
- When a tab closes (a step closes it, or it closes itself, as a sign-in
|
|
1407
|
+
popup does), replay and an exported Playwright test go where the
|
|
1408
|
+
recording's browser went: they stay where they were when another tab
|
|
1409
|
+
closed, and take the tab that slid into the closed one's place, never
|
|
1410
|
+
simply the last tab.
|
|
1411
|
+
- `saffron eject` no longer writes a base URL's password, or a key in its
|
|
1412
|
+
query or fragment (`?api_key=`, `#access_token=`), into
|
|
1413
|
+
`playwright.config.ts` or `saffron-helpers.ts`: it writes `***` and
|
|
1414
|
+
warns, naming the variable to set to the full URL (`BASE_URL`,
|
|
1415
|
+
`API_URL` or `API_URL_<NAME>`).
|
|
1416
|
+
- When a click opens a new tab and the agent keeps working on the page it
|
|
1417
|
+
clicked (the browser it records with stays there), the recording now
|
|
1418
|
+
switches back to that page, so replay runs those steps there instead of
|
|
1419
|
+
on the new tab, where a "not shown" check could pass on the wrong page.
|
|
1420
|
+
A tab closed after such a click is closed from the agent's page too, and
|
|
1421
|
+
a heal that takes over on a tab replay followed records a switch to the
|
|
1422
|
+
tab the agent works on (its browser starts on the first tab). A
|
|
1423
|
+
recording made earlier keeps its behavior: re-record a scenario that
|
|
1424
|
+
opens a tab and then acts on the page that opened it.
|
|
1425
|
+
- Replay follows only a tab a recorded action opened. One that opens
|
|
1426
|
+
without such an action (a popup more than 5 seconds after its click, a
|
|
1427
|
+
page timer) is not followed, as the agent's browser stayed on its page:
|
|
1428
|
+
the checks after it ran on the new tab, where they could pass. Exported
|
|
1429
|
+
tests follow tabs the same way.
|
|
1430
|
+
- An older recording that uses tabs without tab marks still follows any
|
|
1431
|
+
new tab, also one that opens later than 150 ms after a click: the check
|
|
1432
|
+
after it moves to the new tab, and an action waits for the tab or its
|
|
1433
|
+
element, a `{unique:...}` target included. A click that opens no tab is
|
|
1434
|
+
not slowed. A "not shown" check there must hold on the page it started
|
|
1435
|
+
on, then waits for the tab and must hold on it too, so neither page
|
|
1436
|
+
passes it for the other (record the scenario again to add tab marks and
|
|
1437
|
+
drop the wait).
|
|
1438
|
+
- Exported tests wait for a tab a background click opened, and take a tab
|
|
1439
|
+
that already opened before a switch or close, as replay does.
|
|
1440
|
+
- Reports and run history mask a key in the base URL's query or fragment
|
|
1441
|
+
(`?api_key=`, `#access_token=`), as they already masked its password.
|
|
1442
|
+
- A signed storage URL's signature (an Azure SAS `?sig=`) is masked
|
|
1443
|
+
wherever a key in a URL's query is.
|
|
1444
|
+
- `saffron eject` warns when the exported tests send API requests while
|
|
1445
|
+
`transport.api` or `transport.module` is set: they go without its
|
|
1446
|
+
headers, proxy or certificate setting.
|
|
1447
|
+
- A feature path a Windows machine recorded (`features\login.feature`)
|
|
1448
|
+
is found on macOS and Linux: `saffron prune` deleted such a recording of
|
|
1449
|
+
a live feature outside the features folder, `saffron import` refused a
|
|
1450
|
+
Windows CI run's proposals as "not in this checkout", `saffron trace`
|
|
1451
|
+
called them stale, `accept` could not apply their feature edits, and
|
|
1452
|
+
viewing that run hid "Open scenario". `saffron coverage` writes feature
|
|
1453
|
+
paths with forward slashes, and `--base` reads a map committed on
|
|
1454
|
+
Windows as the same files instead of every scenario removed.
|
|
1455
|
+
- Feature paths written on Windows also match elsewhere in the run: a
|
|
1456
|
+
sharded run split across Windows and Linux machines gives every
|
|
1457
|
+
scenario to exactly one shard (some ran twice and some never), `saffron
|
|
1458
|
+
diff <feature file>` finds a proposal filed on Windows, `run --plan`
|
|
1459
|
+
treats such a pending proposal as the run does instead of planning a
|
|
1460
|
+
recording, and `status --json` names an Examples row's scenario
|
|
1461
|
+
(`baseScenario`) from a report written on Windows. `saffron init` keeps
|
|
1462
|
+
the Windows line endings of an AGENTS.md or CLAUDE.md that uses them.
|
|
1463
|
+
- A dropdown option picked in a form is replayed by its label when
|
|
1464
|
+
another option's value is the same text (that earlier option was chosen
|
|
1465
|
+
instead). Exported tests do the same.
|
|
1466
|
+
- Recording with many agent workers no longer prints a
|
|
1467
|
+
MaxListenersExceededWarning: the sessions share one exit listener.
|
|
1468
|
+
- A fixture whose pending proposal is deleted or unreadable mid-run (with
|
|
1469
|
+
proofs off) makes its scenarios red, naming why, instead of ending the
|
|
1470
|
+
run without a report.
|
|
1471
|
+
- `saffron accept` writes a feature edit atomically, keeping the file's
|
|
1472
|
+
line endings, permissions and symlink, so a crash mid-write no longer
|
|
1473
|
+
leaves it cut short.
|
|
1474
|
+
- One StepSet file that does not parse, or defines a set twice, no longer
|
|
1475
|
+
stops `saffron steps` and the MCP `search_steps` and `list_step_sets`
|
|
1476
|
+
tools: the other files' steps are offered and that file is named.
|
|
1477
|
+
- The language server refreshes `{data:...}` hints when files in a
|
|
1478
|
+
configured `dataDir` change, not only in `data/`.
|
|
1479
|
+
- Recording keeps right, middle and Ctrl or Shift clicks: Playwright MCP
|
|
1480
|
+
writes their options over several lines, and they were left out of the
|
|
1481
|
+
recording.
|
|
1482
|
+
- A `{unique:name:digits}` value sent as a number (in a request body or a
|
|
1483
|
+
database check bind) is recorded as its token: it was kept as the
|
|
1484
|
+
recording run's number, so later runs checked that run's row.
|
|
1485
|
+
- An abbreviated amount (`€1.5M`, `$1.2K`), or one written in words or
|
|
1486
|
+
spaced off ("3 bn", "€1.5 million", "1,5 Mio. €", "5 lakh"), is refused
|
|
1487
|
+
instead of read as its digits, and Indian lakh grouping
|
|
1488
|
+
(`₹1,23,456.00`) reads as 123456, in replay and exported tests.
|
|
1489
|
+
- Copies of a request sent at once (`concurrent`) are never sent again by
|
|
1490
|
+
`retries`: a retry could pass a race the first copies lost.
|
|
1491
|
+
- Response bodies are read in the charset their Content-Type names, for
|
|
1492
|
+
API requests, the browser's response checks and exported tests, and a
|
|
1493
|
+
leading byte order mark is ignored: a Latin-1 body read garbled, so a
|
|
1494
|
+
`differs` check could pass on an unchanged value.
|
|
1495
|
+
- A recording the agent's provider failed (a sign-in that expired, the
|
|
1496
|
+
network, a usage limit) is counted as blocked by the agent's provider,
|
|
1497
|
+
not as failed in the app.
|
|
1498
|
+
- Stopping a run (Ctrl-C, a cancelled CI job) also stops a running Codex
|
|
1499
|
+
or Antigravity session, which kept running and spending tokens.
|
|
1500
|
+
- `maxSessionMinutes` also stops a session that makes no tool call (a
|
|
1501
|
+
vendor waiting on a sign-in, a stalled model call), two minutes past its
|
|
1502
|
+
budget, and says why. Such a recording counts as failed, not as blocked
|
|
1503
|
+
by the provider, as the app may be what kept it waiting.
|
|
1504
|
+
- `saffron import` refuses bundles whose manifests, reports and proposals
|
|
1505
|
+
would unzip to more than 1 GB together, before reading any, and masks a
|
|
1506
|
+
password or token in a bundle URL wherever it prints it.
|
|
1507
|
+
- Recording keeps the checkboxes and radio buttons the agent sets while
|
|
1508
|
+
filling a form, a key press of `{`, `(`, `[`, `,`, `'` or `\`, and a
|
|
1509
|
+
navigation to an address with an apostrophe; each was dropped without a
|
|
1510
|
+
warning, and replay never did it.
|
|
1511
|
+
- `saffron trace` refuses a trace file that unzips past its declared size
|
|
1512
|
+
or over 1 GB, so a damaged or hostile trace from CI cannot fill memory.
|
|
1513
|
+
- A "not shown" check passed while the text was on screen with an
|
|
1514
|
+
invisible format character in it: the bidi isolates Fluent and ICU put
|
|
1515
|
+
around a value, a soft hyphen, a word joiner. These now count as shown.
|
|
1516
|
+
- A step seeded from another with a new quoted value wrote it into a
|
|
1517
|
+
pattern unescaped: `matches "Paid (partial)"` passed on "Paid partial".
|
|
1518
|
+
- A text check whose expected value holds a captured credential
|
|
1519
|
+
(`{captured:...}`) is no longer handed to the page's own scripts, as
|
|
1520
|
+
`{env:...}` values already were, so a script on the page or in an iframe
|
|
1521
|
+
cannot read it.
|
|
1522
|
+
- A closed single dropdown reads as its selected option only: a text,
|
|
1523
|
+
pattern, capture or "differs" check no longer passes on another option,
|
|
1524
|
+
and a "not shown" check fails while the dropdown shows the option it
|
|
1525
|
+
names. A text target ("I see Cancelled") never matches an option the
|
|
1526
|
+
dropdown only offers. A list check still reads every option.
|
|
1527
|
+
- A "not shown" check by role and name fails when the page shows that name
|
|
1528
|
+
in another case ("CANCEL ORDER").
|
|
1529
|
+
- Seeding by quoted value keeps every date token form (`{date}`,
|
|
1530
|
+
`{date+1}`, `{date+1:D MMM}`) as written in a pattern.
|
|
1531
|
+
- A multipart request never sends `saffron.config.json` (in any letter
|
|
1532
|
+
case), which may hold a base URL's password.
|
|
1533
|
+
- A CI run's results never offer to open a feature path outside the
|
|
1534
|
+
project.
|
|
1535
|
+
- The language server reads the data environment as a run does: a
|
|
1536
|
+
`SAFFRON_ENV` set in the shell, even empty, keeps the `.env` one out.
|
|
1537
|
+
- A request the page started before an action (a long-poll opened on
|
|
1538
|
+
load) is no longer taken for the action's own when its event arrives a
|
|
1539
|
+
moment late, so "not shown", count and response checks no longer wait it
|
|
1540
|
+
out: four such checks took 14 seconds instead of 2.
|
|
1541
|
+
- A check that a dropdown offers an option reads the option itself
|
|
1542
|
+
(`expectText` on `option "Cancelled"`). A 1.3.x check on the dropdown
|
|
1543
|
+
that passed on one of its other options is now red: record it again.
|
|
1544
|
+
- Under `--no-agent`, a run where the app or the database does not answer
|
|
1545
|
+
marks those scenarios blocked by the environment, as a run with the
|
|
1546
|
+
agent does, so CI tells an outage from a regression.
|
|
1547
|
+
- Colour only on a terminal, and never with `NO_COLOR` set: a log file or
|
|
1548
|
+
a pipe gets plain text.
|
|
1549
|
+
- An exported "not shown" check counts as shown what replay does: a text
|
|
1550
|
+
with invisible format characters in it, a role's name in another case,
|
|
1551
|
+
the selected option of a closed dropdown.
|
|
1552
|
+
- `saffron eject` says to install `@playwright/test` before
|
|
1553
|
+
`npx playwright test`.
|
|
1554
|
+
- A message watched for with "and see" (a toast, a flash, a status) is also
|
|
1555
|
+
watched inside the page when its target is a test id or plain CSS: one
|
|
1556
|
+
shown for a few milliseconds is seen, even on a busy machine or when the
|
|
1557
|
+
page moves on at once, where a look every 50 ms could miss it. A watched
|
|
1558
|
+
text that holds a secret is never handed to the page.
|
|
1559
|
+
- A {unique:...} value the page shows in another case (uppercase through
|
|
1560
|
+
CSS) goes back to its token in the recording, so a "not shown" check
|
|
1561
|
+
never keeps last run's value; a {unique:name:digits} value the page
|
|
1562
|
+
groups ("984,905,203") is refused when the recording is saved, as no
|
|
1563
|
+
token stands for it.
|
|
1564
|
+
- The check after a heal follows only a tab its action was recorded
|
|
1565
|
+
opening, as replay does: a popup a page timer opened mid-check no longer
|
|
1566
|
+
passes the check on another page.
|
|
1567
|
+
- A secret in a URL's query or fragment typed on the command line
|
|
1568
|
+
(`--base-url "https://app.test/?access_token=..."`) is masked in the
|
|
1569
|
+
report's command, the HTML report, the console summary and export
|
|
1570
|
+
bundles, as a URL password already was.
|
|
1571
|
+
- A run whose only reds are scenarios the recorder could not run (a stale
|
|
1572
|
+
or missing recording under --no-agent) no longer reads "all clear" in
|
|
1573
|
+
the HTML report or "0 failed" alone in `saffron status`: both say the
|
|
1574
|
+
run failed.
|
|
1575
|
+
- saffron.config.json refuses a number or other non-string for a path or
|
|
1576
|
+
name setting (baseURL, features, dataDir, storageState, timezone, model
|
|
1577
|
+
and others, and a session's storageState) at load, instead of failing
|
|
1578
|
+
later with a raw Node error after agent work.
|
|
1579
|
+
- `saffron report --merge` exits 1 when the merged run has a red, or a
|
|
1580
|
+
scenario pending review with `strict` on, as `saffron run` does.
|
|
1581
|
+
- `saffron diff` with a proposal file that is not there, and `saffron
|
|
1582
|
+
author` with a requirements file that is not there, say "<file> does not
|
|
1583
|
+
exist" and exit 2 instead of crashing.
|
|
1584
|
+
- A placeholder an Outline's Examples have no column for (a misspelt
|
|
1585
|
+
`<rol>` beside `| role |`), in a step, doc string or table cell, is now
|
|
1586
|
+
an error naming the columns (with inline Examples, only a near miss of a
|
|
1587
|
+
column: other tags such as `"<script>"` or `List<String>` stay
|
|
1588
|
+
literal), instead of every row running the same text and later rows
|
|
1589
|
+
passing on row 1's recording.
|
|
1590
|
+
- An Examples block with no table (`Examples: data:roles`, `Examples:
|
|
1591
|
+
{roles}`, or a table forgotten) is now an error instead of running no
|
|
1592
|
+
row without a word. A header row with no rows yet still runs nothing.
|
|
1593
|
+
- A Scenario Outline with no Examples whose steps use a placeholder is
|
|
1594
|
+
now refused ("add Examples or use Scenario:") instead of running once
|
|
1595
|
+
with its placeholders as literal text.
|
|
1596
|
+
- A `*` step opening a scenario after a Background `Given` is no longer
|
|
1597
|
+
counted as part of the Background's setup.
|
|
1598
|
+
- A .feature file whose description or doc string has a line starting
|
|
1599
|
+
`Fixture:` is no longer told to rename to .saffron. Only a `Fixture:`
|
|
1600
|
+
line with steps under it is.
|
|
1601
|
+
- Editor snippets escape `$`, `\` and `}` in step texts and StepSet names,
|
|
1602
|
+
so a name holding one completes as written.
|
|
1603
|
+
- The AGENTS.md block `saffron init` writes names the features and data
|
|
1604
|
+
folders `saffron.config.json` sets, not always `features/` and `data/`.
|
|
1605
|
+
- `saffron init` reads a VS Code `.vscode/mcp.json` with comments and
|
|
1606
|
+
trailing commas (JSONC) instead of calling it invalid JSON; one already
|
|
1607
|
+
holding the saffron entry is left as is, and one that needs it says what
|
|
1608
|
+
to add by hand rather than drop the comments.
|
|
1609
|
+
- A database check written as count(*) and checked as 0 is now treated as
|
|
1610
|
+
"not in the database": an empty bind is refused, the 0 must keep holding
|
|
1611
|
+
for all of db.timeout, and a schema repair of it is refused.
|
|
1612
|
+
- A schema repair of a database check whose row count comes from an
|
|
1613
|
+
Outline value (a count of `<n>`) is refused, since that value may be 0.
|
|
1614
|
+
- On MySQL, a check that locks rows (FOR UPDATE, FOR SHARE, SKIP LOCKED,
|
|
1615
|
+
LOCK IN SHARE MODE) is refused.
|
|
1616
|
+
- Database writes through a SQL Server synonym or view, or a MySQL view,
|
|
1617
|
+
are refused, since they may reach a table in another database.
|
|
1618
|
+
- On MySQL, INSERT ... SET is read with its columns, as INSERT ... VALUES
|
|
1619
|
+
is.
|
|
1620
|
+
- A token in an API response that no step extracts (an access_token or
|
|
1621
|
+
refresh_token field, a Location carrying #access_token=) no longer
|
|
1622
|
+
reaches the transcript, reports, traces, error lines or the recording
|
|
1623
|
+
agent in clear.
|
|
1624
|
+
- A polled request (until) keeps polling after a lost connection (a
|
|
1625
|
+
reset) or an attempt that timed out, and fails with that error only once
|
|
1626
|
+
the poll's time is up. A stopped run, a DNS failure or a TLS failure
|
|
1627
|
+
ends the poll at once.
|
|
1628
|
+
- The recording agent can no longer record a response-time limit (maxMs)
|
|
1629
|
+
or a poll limit (until.timeoutMs) other than the one the scenario states
|
|
1630
|
+
in figures: the bound must be a duration the step or the checks right
|
|
1631
|
+
after it name, in any unit.
|
|
1632
|
+
- A Then whose only check is the 2xx class, or the very status its
|
|
1633
|
+
request already required, is refused as a check that could never fail:
|
|
1634
|
+
the agent records a field or schema check instead.
|
|
1635
|
+
- Two requests at once in a new API session no longer open two clients,
|
|
1636
|
+
one of which was never closed.
|
|
1637
|
+
- In a parallel run with the agent, a scenario whose replay failed and
|
|
1638
|
+
then passed when run again for the agent is red, as with one worker: it
|
|
1639
|
+
was reported green, and a failure that comes and goes passed with
|
|
1640
|
+
nothing in the report.
|
|
1641
|
+
- `saffron run --plan` refuses an invalid `--workers` or
|
|
1642
|
+
`--agent-workers`, as a run does.
|
|
1643
|
+
- A failed check on a field named as a credential (a token, a session id)
|
|
1644
|
+
says `[credential, not kept]` instead of its value, in the error, the
|
|
1645
|
+
report and a poll's last reason.
|
|
1646
|
+
- Hiding a credential field in a response keeps the rest of the body as
|
|
1647
|
+
the API wrote it: large ids, decimals like 1.10 and spacing are not
|
|
1648
|
+
rewritten.
|
|
1649
|
+
- Exported response checks wait for the page's requests to settle and fail
|
|
1650
|
+
when the same request is answered again with a status the check rejects,
|
|
1651
|
+
as replay does.
|
|
1652
|
+
- Exported "URL does not contain", count and differs checks hold only once
|
|
1653
|
+
the page settles, as replay's do: a navigation on its way, a list still
|
|
1654
|
+
growing or a value that changes for a moment no longer passes them.
|
|
1655
|
+
- Exported captures and differs checks read the displayed copy of an
|
|
1656
|
+
element after a hidden one with the same text, as replay does.
|
|
1657
|
+
- Exported value checks require a shown input that holds the value in the
|
|
1658
|
+
same look.
|
|
1659
|
+
- Exported "not shown" checks on a literal name pattern also find the name
|
|
1660
|
+
with invisible format characters in it, as replay does.
|
|
1661
|
+
- The exported helper file writes the project root with forward slashes on
|
|
1662
|
+
Windows, so it reads on macOS and Linux too.
|
|
1663
|
+
- A heal of a step that failed partway through is told which of the step's
|
|
1664
|
+
actions replay already performed, and records only the rest; a healed
|
|
1665
|
+
recording no longer repeats them (an Add click run twice before
|
|
1666
|
+
Proceed).
|
|
1667
|
+
- Behavior change: a heal that changes nothing in the recording no longer
|
|
1668
|
+
files a proposal when its proof passes. Such a run used to be a yellow
|
|
1669
|
+
proposal and is now red, saying the recording failed on replay and then
|
|
1670
|
+
passed again (timing or the app's state), as it already was with
|
|
1671
|
+
--no-verify.
|
|
1672
|
+
- A proposal not filed on this machine no longer turns a failed committed
|
|
1673
|
+
recording yellow when it checks the Then steps differently from the
|
|
1674
|
+
committed recording: the run is red and names the pending file.
|
|
1675
|
+
- A check in a browser session that has no page yet (As "Bob") is checked
|
|
1676
|
+
on that session's own page, as replay does, never on the default
|
|
1677
|
+
session's page.
|
|
1678
|
+
- A seeded step that an agent segment went past and recorded nothing for
|
|
1679
|
+
keeps its seeded recording instead of an empty one.
|
|
1680
|
+
- "Checks proven sensitive" is claimed only when every Then was probed; a
|
|
1681
|
+
Then the sensitivity check leaves out (one that clicks, for example)
|
|
1682
|
+
leaves it unclaimed.
|
|
1683
|
+
- A row value checked through XPath `normalize-space(...)`, `string(...)`
|
|
1684
|
+
or `./text()`, or through an unquoted CSS attribute value
|
|
1685
|
+
(`[data-k*=x-<id>]`), is kept whole, so row 1 no longer passes or clicks
|
|
1686
|
+
on "Order 12". A row value inside another XPath function, such as
|
|
1687
|
+
`translate` or `concat`, is refused instead of matched loosely.
|
|
1688
|
+
- A "not shown" check whose selector holds the text itself
|
|
1689
|
+
(`:has-text(...)`, `text=...`, XPath `contains(., ...)`) fails when the
|
|
1690
|
+
page shows that text with invisible characters around the row value,
|
|
1691
|
+
such as the bidi isolates Fluent and ICU add.
|
|
1692
|
+
- An invisible character (soft hyphen, zero-width space, bidi isolate) no
|
|
1693
|
+
longer counts as a word boundary, so "saved" does not pass on "unsaved"
|
|
1694
|
+
written with a soft hyphen, and row 1 does not pass on "Order 12" with a
|
|
1695
|
+
zero-width space inside.
|
|
1696
|
+
- A recorded response, attribute, style or JSON field pattern that
|
|
1697
|
+
backtracks fails the step at once with a message saying so, instead of
|
|
1698
|
+
freezing the run.
|
|
1699
|
+
- A "not shown" check treats straight and typographic quotes alike, so
|
|
1700
|
+
"Can't save" fails while the page shows "Can’t save".
|
|
1701
|
+
- Exported Playwright tests fail a recorded pattern that backtracks (a
|
|
1702
|
+
URL, a response body or JSON field, an attribute, a style) saying so, as
|
|
1703
|
+
replay does, no longer take an invisible format character as a word
|
|
1704
|
+
boundary, and read straight and typographic quotes alike in "not shown"
|
|
1705
|
+
checks.
|
|
1706
|
+
- A credential in an API base URL's query (`api.baseURL` or `--api-url`
|
|
1707
|
+
with `?api_key=`) is masked in request errors, the transcript and what
|
|
1708
|
+
the recording agent reads, as the URL's password already was; one in the
|
|
1709
|
+
browser `baseURL` query (`?token=`) is masked in traces too.
|
|
1710
|
+
- A failed header check never quotes a credential header's value
|
|
1711
|
+
(Set-Cookie) and masks a token in a Location it quotes.
|
|
1712
|
+
- A row value inside an XPath `or` (`contains(., "Order <id>") or
|
|
1713
|
+
contains(., "Order #<id>")`) is refused as one replay cannot tie to its
|
|
1714
|
+
row, as inside `translate` or `concat`: a "not shown" check passed on
|
|
1715
|
+
the other branch.
|
|
1716
|
+
- A "not shown" check through `:text(...)`, `:text-is(...)`,
|
|
1717
|
+
`:text-matches(...)`, XPath `normalize-space(.)="..."` or
|
|
1718
|
+
`text()="..."`, or an exact `internal:text="..."s` also finds the text
|
|
1719
|
+
with invisible characters around the row value.
|
|
1720
|
+
- A numeric one-time code or PIN in an API response (`"otp": 123456`) is
|
|
1721
|
+
hidden in the transcript and error lines, as a text one is.
|
|
1722
|
+
- A recorded pattern bounds a row value before `.*`, `[^,]*` or `.{0,9}`
|
|
1723
|
+
and more text: `/Order <id>.*Shipped/` for row 1 passed on "Order 12
|
|
1724
|
+
Shipped". An optional or loose point after the value (`<id>\.?`,
|
|
1725
|
+
`<id>[,;]`) no longer lets "1.5" or "1,000" pass for 1.
|
|
1726
|
+
- "Not shown" finds text in markup with no space between its elements
|
|
1727
|
+
(React, Vue, minified HTML): "Order 1" passed as gone while the row read
|
|
1728
|
+
"Order 1Pending", and "Bread" while it read "BreadIn stock". A "not
|
|
1729
|
+
shown" by role and name on a role whose name never comes from its text (a
|
|
1730
|
+
list item, a status, an alert, a paragraph) reads its text: it passed
|
|
1731
|
+
with the text still shown.
|
|
1732
|
+
- An attribute check on `checked`, `selected` or `value` reads the control
|
|
1733
|
+
as it is now, in replay and in exported tests: a box the app unticked
|
|
1734
|
+
passed as ticked.
|
|
1735
|
+
- Patterns, captures and amounts never read across a block's edge, as text
|
|
1736
|
+
checks and exported tests do: "Qty 1" over "0 left" no longer reads "Qty
|
|
1737
|
+
10", nor "$1" over "5 nights" the amount 15. A price split into flex items
|
|
1738
|
+
("$" "12" ".50") still reads "$12.50".
|
|
1739
|
+
- A row value written unquoted inside an XPath function
|
|
1740
|
+
(`contains(@data-order, <id>)`) is refused: row 1 found order 12.
|
|
1741
|
+
- A row's number is never written as `<column>` inside a longer number
|
|
1742
|
+
("9.5" in "€9.50", "104" in "order-000104"): another row's "not shown"
|
|
1743
|
+
looked for "€12.250" and passed. A zero-padded id in a locator now binds
|
|
1744
|
+
the recording to its row, so other rows are red until the step names the
|
|
1745
|
+
id as the page writes it.
|
|
1746
|
+
- A two-column table with a repeated first cell is judged whole, so an
|
|
1747
|
+
edit to an earlier row re-records and seeds no other scenario.
|
|
1748
|
+
- A long number an Outline row's page sets in its own way ("+4712345678"
|
|
1749
|
+
shown "+47 123 45 678", an IBAN in blocks) binds the recording to its
|
|
1750
|
+
row, with guidance, where another row passed on it.
|
|
1751
|
+
- The recording agent cannot check a value against one captured in the same
|
|
1752
|
+
Then, nor a response by a pattern every URL matches, nor bind a database
|
|
1753
|
+
check to part of a row's value ("ann" of `ann@example.test`). A watch armed
|
|
1754
|
+
in a step replay ran before the session can be checked.
|
|
1755
|
+
- A recording the post-process refuses is told to the agent and ends as an
|
|
1756
|
+
ordinary red, its cost counted, not as the provider failing.
|
|
1757
|
+
- An API request whose path segment comes out blank (`/orders/{captured:id}`
|
|
1758
|
+
with an empty id) is refused, in replay and in exported tests: it went to
|
|
1759
|
+
the collection URL and passed. A credential a response echoes under a
|
|
1760
|
+
plain name is kept from other origins.
|
|
1761
|
+
- Database checks refuse MySQL `||` (run as OR), a WITH that selects a bind
|
|
1762
|
+
itself, and on PostgreSQL a backslash before a quote; "at most 0" on a
|
|
1763
|
+
count is an absence check, as "equals 0" is.
|
|
1764
|
+
- A kept trace no longer holds the saved login (`storageState`), basic-auth
|
|
1765
|
+
credentials or extra headers, and every cookie in its network log is
|
|
1766
|
+
redacted whatever its name. A `{refresh, access}` token pair's access
|
|
1767
|
+
token is a credential: hidden from the agent, the transcript and check
|
|
1768
|
+
failures, and kept from other origins.
|
|
1769
|
+
- A pinned contract applies `required` on a `readOnly` property to
|
|
1770
|
+
responses only and on a `writeOnly` one to requests only, so one shared
|
|
1771
|
+
model no longer refuses every request body. A response the contract
|
|
1772
|
+
gives as `*/*` is checked against its schema.
|
|
1773
|
+
- `saffron status`, `saffron coverage` and the editors keep the last run's
|
|
1774
|
+
result of each scenario when the report came from a CI machine on another
|
|
1775
|
+
OS.
|
|
1776
|
+
- `--rerecord` puts back a recording it removed when the scenario was not
|
|
1777
|
+
recorded again (the app down, a capability off, the budget spent,
|
|
1778
|
+
Ctrl-C). `run --plan --rerecord` replays a fixture, as the run does.
|
|
1779
|
+
- `saffron report --merge --strict` exits 1 on a pending review, as shards
|
|
1780
|
+
run with `--strict` do.
|
|
1781
|
+
- A parallel run's kept red no longer names evidence its second try
|
|
1782
|
+
deleted.
|
|
1783
|
+
- A feature edit that would drop an Outline `<column>` is skipped, and a
|
|
1784
|
+
file with mixed line endings keeps every other line's ending.
|
|
1785
|
+
- A short config key of the team's own (`app`, `id`) is ignored, not
|
|
1786
|
+
refused as a misspelling; a short key's letters swapped (`evn`) still is.
|
|
1787
|
+
- `differs from` compares `$5` and `5 USD` as one currency.
|
|
1788
|
+
|
|
1789
|
+
Editors (VS Code extension 0.8.2, JetBrains plugin 0.8.2)
|
|
1790
|
+
|
|
1791
|
+
- Database steps: a step recorded with a database check or write is not
|
|
1792
|
+
offered as a zero-token wording to reuse (completion, the step hover and
|
|
1793
|
+
StepSet expansion say it replays at zero tokens only in its own
|
|
1794
|
+
scenario), and a step is marked as one only when every recording of it
|
|
1795
|
+
reads or writes the database.
|
|
1796
|
+
- A proposal with the runner's new "database check repaired after a schema
|
|
1797
|
+
change:" or "database write added by the heal:" note is held from Accept
|
|
1798
|
+
All, as `saffron accept --all` skips it.
|
|
1799
|
+
- The last run's line, a CI run's totals and its rows count a red the app
|
|
1800
|
+
did not cause (a stale recording without the agent, a missing
|
|
1801
|
+
environment, the agent or the budget) as not run, as the runner's status
|
|
1802
|
+
does.
|
|
1803
|
+
- Open trace finds a CI scenario whose feature path the runner left out,
|
|
1804
|
+
by its name.
|
|
1805
|
+
- Eject's This file runs in the open file's own Saffron project, with
|
|
1806
|
+
forward slashes on Windows, and is not offered for a file outside every
|
|
1807
|
+
project.
|
|
1808
|
+
- VS Code: a status or diff that times out is stopped with its child
|
|
1809
|
+
processes, a cut diff says so, and `{data:...}` hints read the
|
|
1810
|
+
environment as the runner does. JetBrains: the Dashboard reloads only on
|
|
1811
|
+
new runner output, and proposals and lenses no longer read the disk on
|
|
1812
|
+
the UI thread.
|
|
1813
|
+
|
|
8
1814
|
## 1.3.5: Scenarios kept from running at once
|
|
9
1815
|
|
|
10
1816
|
Released 2026-11-08.
|
|
@@ -1606,7 +3412,7 @@ Runner
|
|
|
1606
3412
|
newest marked. A duration over a minute reads as minutes and seconds,
|
|
1607
3413
|
and scenarios sharing a cause get a button for the command that deals
|
|
1608
3414
|
with them all, the command itself still on the page.
|
|
1609
|
-
- **A compatibility page per release.** [Compatibility](COMPATIBILITY.md)
|
|
3415
|
+
- **A compatibility page per release.** [Compatibility](docs/COMPATIBILITY.md)
|
|
1610
3416
|
lists the Node.js, operating systems, Playwright and browsers, each
|
|
1611
3417
|
agent's SDK (the range accepted and the version tested) and the recording
|
|
1612
3418
|
versions this release reads and writes, generated from the release's own
|
|
@@ -1618,7 +3424,7 @@ Runner
|
|
|
1618
3424
|
greens); `--record` also records the suite afresh with the agent and
|
|
1619
3425
|
measures the share verified on the first attempt, the share held for
|
|
1620
3426
|
review, and the time and cost. The results are published in the docs
|
|
1621
|
-
([benchmark](BENCHMARK.md)): today 66 of 66 replays green and no false
|
|
3427
|
+
([benchmark](docs/BENCHMARK.md)): today 66 of 66 replays green and no false
|
|
1622
3428
|
green on 8 broken apps.
|
|
1623
3429
|
- **A coverage map per pull request.** `saffron coverage` maps each
|
|
1624
3430
|
requirement ID (a tag matching `requirementTag`, such as `@REQ-12`) to
|
|
@@ -2383,7 +4189,7 @@ Runner
|
|
|
2383
4189
|
("red" in "Delivered", "Pro" in "Products") no longer binds the row; a
|
|
2384
4190
|
camelCase part ("SelectedRed") still does. Rows whose data reads blank
|
|
2385
4191
|
on every row no longer record each other forever. A row value holding a
|
|
2386
|
-
secret's value (
|
|
4192
|
+
secret's value (`admin@shop.test` beside a variable "admin") is the
|
|
2387
4193
|
row's `<column>`, and a secret equal to a row's lookup key leaves the
|
|
2388
4194
|
step's `{data:...}` reference alone. A state check ("Promo" hidden, an
|
|
2389
4195
|
attribute's pattern) binds the row whatever its element is named. A
|