saffron-ai 1.4.0 → 1.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +1159 -2
- package/README.md +7 -5
- package/dist-pkg/cli.js +473 -451
- package/package.json +27 -6
- package/skills/saffron/SKILL.md +40 -18
- package/skills/saffron/references/api.md +3 -1
- package/skills/saffron/references/config.md +1 -1
- package/skills/saffron/references/database.md +173 -10
- package/skills/saffron/references/syntax.md +35 -12
- package/templates/playwright/saffron-helpers.ts +608 -110
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,1163 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.4.1: Database steps on seven more databases
|
|
9
|
+
|
|
10
|
+
Released 2026-11-17.
|
|
11
|
+
|
|
12
|
+
Runner
|
|
13
|
+
|
|
14
|
+
- Database steps on six more databases (beta), beside MongoDB below, each
|
|
15
|
+
tested against the real database or its emulator:
|
|
16
|
+
- Oracle (`oracle://user:pass@host:1521/service`, `npm install oracledb@6
|
|
17
|
+
node-sql-parser`; the driver's thin mode needs no Oracle client). A
|
|
18
|
+
statement is read as plain SQL: `FETCH FIRST`, `OFFSET` and `ROWNUM`
|
|
19
|
+
count as a `LIMIT`, and `CONNECT BY`, `(+)`, flashback, `q'...'`
|
|
20
|
+
strings, `MERGE` and PL/SQL are refused. Reads run in a `READ ONLY`
|
|
21
|
+
transaction; a write is read back by its primary key, the key Oracle
|
|
22
|
+
gave an inserted row read from that row itself. Names are in capitals,
|
|
23
|
+
so a row's columns are checked as `/0/NAME`.
|
|
24
|
+
- SQLite (`sqlite:path/to/app.db`, `npm install node-sql-parser`; the
|
|
25
|
+
driver is Node's own, from Node 22.13). The file is opened read-only
|
|
26
|
+
for a check and never created. `INSERT OR REPLACE` and the like are
|
|
27
|
+
refused. A "not in the database" check that only a type keeps empty
|
|
28
|
+
(`json_extract(...) = $1` with text where the JSON holds a number) is
|
|
29
|
+
refused. `db.allow` lists the file's name or path.
|
|
30
|
+
- MariaDB (`mariadb://` or `mysql://`). It runs as MySQL, and did not
|
|
31
|
+
before: every statement failed on a session setting MariaDB does not
|
|
32
|
+
have, and cleanup put a `JSON` column back respelt (`1.10` as `1.1`).
|
|
33
|
+
- CockroachDB (`cockroachdb://` or `postgres://`). It runs as PostgreSQL,
|
|
34
|
+
and did not before: every write failed, as CockroachDB does not say
|
|
35
|
+
which table a written row is in. A table with no primary key (a hidden
|
|
36
|
+
`rowid`) counts as having none, and a read the database asks to run
|
|
37
|
+
again is read again.
|
|
38
|
+
- DynamoDB (`dynamodb://<region>`, or `dynamodb://<host>:<port>` for
|
|
39
|
+
DynamoDB Local; `npm install @aws-sdk/client-dynamodb`). A step holds
|
|
40
|
+
one JSON command, as on MongoDB: `get`, `query`, `put` or `update`,
|
|
41
|
+
with no scan and no delete. A key attribute is sent as the type the
|
|
42
|
+
table declares; a "not in the database" check is red when an item is
|
|
43
|
+
found with a bind sent as another type, and never reads a global
|
|
44
|
+
secondary index. A `put` never replaces an item. Tested on DynamoDB
|
|
45
|
+
Local; AWS itself is not part of the test suite.
|
|
46
|
+
- Firestore (`firestore://<project-id>`, or
|
|
47
|
+
`firestore://<host>:<port>/<project-id>` for the emulator; `npm
|
|
48
|
+
install @google-cloud/firestore@7`). One JSON command per step: `get`,
|
|
49
|
+
`query`, `create`, `add` or `update`. Placeholders name Firestore's
|
|
50
|
+
types (`"$1:timestamp"`, `"$1:reference"`); a "not in the database"
|
|
51
|
+
check is red when a document is found with a bind sent as another
|
|
52
|
+
type. A `create` never writes over a document. Tested on the
|
|
53
|
+
emulator; Google Cloud itself is not part of the test suite.
|
|
54
|
+
- What tells AI agents how to use Saffron is brought up to this release:
|
|
55
|
+
the authoring skill and its references (the new databases and how to
|
|
56
|
+
word a step for each, exact headers and keywords, `--filter` with
|
|
57
|
+
`and`, `or` and `not`, amounts compared to the decimals shown), the
|
|
58
|
+
block `saffron init` writes into `AGENTS.md` and `CLAUDE.md`, and the
|
|
59
|
+
MCP server's `project_status`, which names the database's engine and
|
|
60
|
+
how its steps are worded. Run `npx saffron init` again to take the new
|
|
61
|
+
block and skill.
|
|
62
|
+
- A recording with a database step on one of the new engines is written
|
|
63
|
+
as schema version 22, so an older runner asks for an update instead of
|
|
64
|
+
a re-record it could not make.
|
|
65
|
+
- Database steps on MongoDB (beta): set `db.url` to a `mongodb://` or
|
|
66
|
+
`mongodb+srv://` URL that names its database, and `npm install mongodb`
|
|
67
|
+
(no SQL parser needed). A step holds one JSON command where a SQL step
|
|
68
|
+
holds its statement: `{"find": "customers", "filter": {"email": "$1"}}`
|
|
69
|
+
or `{"count": ...}` for a check, `{"insert": ..., "documents": [...]}`
|
|
70
|
+
or `{"update": ..., "filter": {...}, "set": {...}}` for a write. The
|
|
71
|
+
rules are the SQL ones: every value a placeholder bound from the step
|
|
72
|
+
(typed where the field needs it: `"$1:objectId"`, `"$1:uuid"`,
|
|
73
|
+
`"$1:long"`, `"$1:decimal"`, `"$1:date"`), a check tied to a value the
|
|
74
|
+
scenario wrote, filters limited to comparison operators (`$where`,
|
|
75
|
+
`$expr`, `$regex` and aggregation are refused), and cleanup that deletes
|
|
76
|
+
inserted documents and puts back, type for type, only the paths an
|
|
77
|
+
update set. A "not in the database" check is judged again with each
|
|
78
|
+
bind sent as each other type it reads as (text, ObjectId, UUID, number,
|
|
79
|
+
decimal, long, date), at its start and at the end of its window: a
|
|
80
|
+
document found so is red, as is a field no document has (compared with
|
|
81
|
+
a bind, in a range, under `$elemMatch` or `$exists` true) or a
|
|
82
|
+
collection that is not there. A UUID reads as its dashed text. A
|
|
83
|
+
whole document or list compared as a value is refused (compare its
|
|
84
|
+
fields with dotted paths), and so is a projection beside a check that a
|
|
85
|
+
field is missing. An update that fails part way puts back what it
|
|
86
|
+
changed first. A failed MongoDB check is never repaired. Recordings
|
|
87
|
+
with MongoDB steps are cache version 21, so an older runner asks to be
|
|
88
|
+
updated instead of advising a re-record it cannot do.
|
|
89
|
+
- On every engine, an `every` check under `LIMIT`, `TOP` or `DISTINCT`
|
|
90
|
+
is refused, as a count is: it cannot see a second, wrong row.
|
|
91
|
+
- A value extracted from inside a credential object (`/session/id`,
|
|
92
|
+
`/session/0/id`, `/credentials/access`, kept as `ref`) is now a
|
|
93
|
+
credential too when it is a text of 6 characters or more: it is masked,
|
|
94
|
+
and never sent to another origin, as the body already showed it hidden
|
|
95
|
+
(`/auth/role` holding "admin" stays plain). So is a credential inside an
|
|
96
|
+
extracted object (`extract "/data"` holding `session: { id }`), a list
|
|
97
|
+
under such an object (`auth.backup[0]`), the texts of a list or object
|
|
98
|
+
whose name ends in `tokens` (`refreshTokens`, while `total_tokens`
|
|
99
|
+
holding a number stays plain), and a name in the plural (`apiKeys`,
|
|
100
|
+
`passwords`). Before, the
|
|
101
|
+
session id or token could reach a second host. Exported tests read
|
|
102
|
+
credentials the same way.
|
|
103
|
+
- Postgres: `db.cleanup` now puts back a json or jsonb column whose change
|
|
104
|
+
was only past 15 digits (`9007199254740992` to `9007199254740993`).
|
|
105
|
+
Both sides of the comparison were read as JavaScript numbers, so the
|
|
106
|
+
change looked like none and the new value stayed. A write's `extract`
|
|
107
|
+
of such a number now keeps every digit instead of a neighbouring id.
|
|
108
|
+
- An API scenario's database checks now read again at `pollIntervalMs`,
|
|
109
|
+
as a browser scenario's do, instead of every 250 ms.
|
|
110
|
+
- An interaction that ran is no longer run again by `retries`: an action
|
|
111
|
+
recorded as opening a tab when none opens, a click armed for a download
|
|
112
|
+
or a file chooser that starts none, a click or key press whose page load
|
|
113
|
+
outlasts the action timeout, and a checkbox the click did not change all
|
|
114
|
+
fail at once. Before, a retry clicked again, and a submit could send its
|
|
115
|
+
order twice (and, on a reloaded form, pass). A `newTab` whose page fails
|
|
116
|
+
to load is closed, so a retry's tab is the only one.
|
|
117
|
+
- A response header sent more than once is checked as it means: each
|
|
118
|
+
Set-Cookie on its own (of several cookies, a pattern names its cookie
|
|
119
|
+
first, `^session=.*HttpOnly`), and a header that holds one value
|
|
120
|
+
(X-Frame-Options, Content-Type, Location) with every value, so a `DENY`
|
|
121
|
+
beside an `ALLOWALL` is red. A list-valued header (Vary, Cache-Control)
|
|
122
|
+
is still read joined. Exported tests check headers the same way.
|
|
123
|
+
- A JSON check whose path holds an empty row value (`/stock/<id>` with an
|
|
124
|
+
empty cell) is refused instead of reading a field no response has, where
|
|
125
|
+
"not there" passed on anything. A check typed as a string fails on a big
|
|
126
|
+
number, so an id the API turned from text into a number is red.
|
|
127
|
+
- Recordings that may turn red with this release, and need
|
|
128
|
+
`saffron run --rerecord`: a check typed as a string on a big number
|
|
129
|
+
(the old advice for such numbers; check it with no type), and a
|
|
130
|
+
Set-Cookie pattern that names no cookie on a response that sets several.
|
|
131
|
+
- Credentials in a form-encoded answer (an OAuth token endpoint's
|
|
132
|
+
`access_token=...&scope=...`) and in JSON after an anti-XSSI prefix are
|
|
133
|
+
hidden from the agent, errors and transcripts.
|
|
134
|
+
- An Examples table with a header row and no rows is now an error. Before,
|
|
135
|
+
the outline ran nothing, so deleting its last row removed it from the
|
|
136
|
+
run while CI stayed green. Comment a draft out until it has a row.
|
|
137
|
+
- `--max-session-minutes` (`maxSessionMinutes`) now limits API recording
|
|
138
|
+
and healing sessions too, and they print their progress as browser ones
|
|
139
|
+
do.
|
|
140
|
+
- A header pattern (`expectHeader`) or a downloaded file's pattern that
|
|
141
|
+
backtracks (`(a+)+$`) now fails the check instead of freezing the run,
|
|
142
|
+
as JSON and page patterns already did.
|
|
143
|
+
- A JSON data file with a number a JavaScript number reads as another
|
|
144
|
+
value (an id past 2^53, more digits than a double holds, `1e400` read
|
|
145
|
+
as Infinity or `1e-400` as 0) is now an error that asks for it in
|
|
146
|
+
quotes, instead of sending and checking another value.
|
|
147
|
+
- A feature file whose scenario header Gherkin does not know (`scenario:`,
|
|
148
|
+
`Scenario :`, `Scenario Name` with no colon, another language's keyword
|
|
149
|
+
under an English file) is now an error. Gherkin reads such a line, and
|
|
150
|
+
the steps under it, as the feature's description, so the scenario, or
|
|
151
|
+
the whole file, left the run without a word while the suite stayed
|
|
152
|
+
green. So is a step above a scenario's first step that Gherkin cannot
|
|
153
|
+
read as one (a tab or a no-break space after its keyword, an invisible
|
|
154
|
+
character before it), a `StepSet` call there in a `.feature` file, and
|
|
155
|
+
`stepset` in the wrong case: each was left out of its scenario. Prose is
|
|
156
|
+
left alone: one description line that starts with "When", or a wrapped
|
|
157
|
+
line that starts with "and", stays description. The language server
|
|
158
|
+
reports these, and an Examples table with no rows or no table, in every
|
|
159
|
+
editor that uses it.
|
|
160
|
+
- A recording is recorded again when a step reads a table value another
|
|
161
|
+
step gave it (`<table:name>`) and an edit leaves two tables disagreeing
|
|
162
|
+
on that value. Before, the token resolved to nothing, replay looked for
|
|
163
|
+
the text `<table:name>` itself, and a "not shown" check passed on it
|
|
164
|
+
with the value on the page.
|
|
165
|
+
- A "not shown" check ending in a row's whole number (`<amt>` holding 10)
|
|
166
|
+
now counts `10.00` on the page as shown. It passed as gone.
|
|
167
|
+
- A "not shown" check in a step that names no column, holding a word of
|
|
168
|
+
the row's value in another order or spelling ("Lee, Ann" for the guest
|
|
169
|
+
"Ann Lee"), now binds its recording to that row, so the other rows fail
|
|
170
|
+
with guidance instead of asking whether the first row's guest is gone.
|
|
171
|
+
- An amount check compares to the unit the amount is shown with: a rate
|
|
172
|
+
of 0.125 no longer passes on 0.129. Amounts shown with two decimals or
|
|
173
|
+
fewer still compare to the cent, so a tax worked out as 7.9992 is the
|
|
174
|
+
8.00 shown. Two captures that differ under a cent (0.125 and 0.129) now
|
|
175
|
+
count as differing. The unit is read from the text shown, so a rate
|
|
176
|
+
shown as 0.1200 does not pass on 0.1204, nor 0.0000002 on 0.0000001.
|
|
177
|
+
- `saffron accept --all` now holds a `Then` recorded again with another
|
|
178
|
+
pattern (`^Paid$` become `^Pending$` or `^.*$`), another attribute or
|
|
179
|
+
style property, another response URL pattern, a URL check turned from
|
|
180
|
+
"contains" to "excludes", or another compared capture, under "check
|
|
181
|
+
weakened:". Only values, statuses, URLs and JSON checks were compared.
|
|
182
|
+
The same pattern with an anchor gained, or grouped another way, is not
|
|
183
|
+
held.
|
|
184
|
+
- Database checks: a query whose WHERE picks one of its table's own rows
|
|
185
|
+
by a subquery (`WHERE id = (SELECT max(id) FROM orders WHERE ref = $1)`,
|
|
186
|
+
`created_at >= ALL (...)`) is read as cutting the rows down, as `LIMIT 1`
|
|
187
|
+
is: a count of the rows or "no second row" under it is refused, and a
|
|
188
|
+
repair may not add it. A subquery on another table is left alone.
|
|
189
|
+
- Database checks: a "not in the database" check refuses a bind of spaces
|
|
190
|
+
alone, as an empty one, and a bind that is a whole number past 2^53,
|
|
191
|
+
which a recording had already rounded to a neighbouring id (bind it as
|
|
192
|
+
text).
|
|
193
|
+
- Database rows shown in a failed check and in the report hide a column
|
|
194
|
+
named as a credential (`api_token`, `password_hash`), as a response
|
|
195
|
+
body's fields are hidden. `SELECT *` on a users table put its hashes and
|
|
196
|
+
tokens in the report. A name ending in `hash` or `digest` after a
|
|
197
|
+
credential's (`passwordHash`, `token_digest`) now names a credential
|
|
198
|
+
everywhere.
|
|
199
|
+
- A repair of a database check bound to a number an earlier step kept
|
|
200
|
+
(an id) is no longer refused: the recorded bind carries its type, which
|
|
201
|
+
a repair cannot write, so the two never compared equal. The recorded
|
|
202
|
+
bind is the one kept, and the one the repair is checked with on the
|
|
203
|
+
live data before its proposal is filed.
|
|
204
|
+
- MongoDB, beyond the first entry above: a whole document or list compared under
|
|
205
|
+
`$eq`, `$ne`, `$in`, `$nin` or `$all` is refused, as it is written
|
|
206
|
+
plainly; a projection or a set path through `$`, `$[]` or `$[name]` is
|
|
207
|
+
refused (name a list's item by its number, `items.0.qty`); `$0` is no
|
|
208
|
+
placeholder; a `:date` that is no day of the calendar (`2026-02-30`,
|
|
209
|
+
hour 24) is refused instead of read as the next day. A "not in the
|
|
210
|
+
database" check is judged on each bound field alone as well, so two
|
|
211
|
+
binds sent as the wrong type at once no longer pass, and a check that a
|
|
212
|
+
field is missing is refused under a projection at any depth. A write
|
|
213
|
+
whose documents cannot be read back is undone at once (inserted
|
|
214
|
+
documents deleted, set paths put back) instead of left for a cleanup
|
|
215
|
+
that never heard of it. An update that set an item past the end of a
|
|
216
|
+
list puts the list back whole, also when it changed an item of that
|
|
217
|
+
list in the same update, and a path named as an object's own
|
|
218
|
+
member (`constructor`) is removed again.
|
|
219
|
+
- MySQL: options in `db.url`'s query can no longer turn the adapter's own
|
|
220
|
+
off. `?multipleStatements=true` or `?decimalNumbers=true` in a URL
|
|
221
|
+
shared with the app won over them.
|
|
222
|
+
- Exported tests hide a credential field in a response body they quote
|
|
223
|
+
and never quote a credential header's value, and read a token pair's
|
|
224
|
+
access token as a credential, all as replay does.
|
|
225
|
+
- The reset commands (`reset` in the config) no longer run before a
|
|
226
|
+
pending proposal is only replayed: every later Examples row, and a
|
|
227
|
+
scenario waiting for review, reset the project's data and then recorded
|
|
228
|
+
nothing. They run when a recording really follows.
|
|
229
|
+
- `saffron author` with the Antigravity provider no longer fails at "the
|
|
230
|
+
session's turn limit" after the draft came back.
|
|
231
|
+
- In a heal, an assert, a watch or a network wait the agent makes before
|
|
232
|
+
its first step, or under a step that had run, is answered as not
|
|
233
|
+
recorded, with why. It was answered "Assertion recorded.", and surfaced
|
|
234
|
+
only as a refused finish.
|
|
235
|
+
- `saffron diff --html <file>` no longer writes the review page over the
|
|
236
|
+
file named after it. `--html` takes its file only as `.html`: a proposal
|
|
237
|
+
or feature file there is a file to review, and the page goes to its
|
|
238
|
+
default place. Before, a recording that cost AI usage, or a feature
|
|
239
|
+
file, was replaced by the page.
|
|
240
|
+
- `saffron report --merge` folds the last report's rows in only for
|
|
241
|
+
shards of a run that named its scenarios (`file:line`, `file:name`).
|
|
242
|
+
Shards of a `--filter` or path run were read the same way, so
|
|
243
|
+
`latest.json` held another run's scenarios and reds.
|
|
244
|
+
- `saffron status` (and the dashboard and the editors' panels) reads a
|
|
245
|
+
last report whose rows are not rows as no report, instead of crashing.
|
|
246
|
+
- `saffron eject` refreshes its own specs when they have CRLF line
|
|
247
|
+
endings (a Windows checkout), instead of keeping them for good as
|
|
248
|
+
edited.
|
|
249
|
+
- `saffron accept --all` and `reject --all` exit 1, saying it once, when
|
|
250
|
+
a pending proposal cannot be read.
|
|
251
|
+
- Config: `actionTimeoutMs: 0` is refused (Playwright reads 0 as no
|
|
252
|
+
timeout; here every check timed out at once), and so is a `transport`
|
|
253
|
+
that is not an object or `capabilities` given as a list.
|
|
254
|
+
- `--filter` refuses `&&`, `!` and `~` from other tools instead of reading
|
|
255
|
+
them as tags: `"@a && @b"` ran `@a` or `@b`.
|
|
256
|
+
- An Outline with thousands of rows (an Examples data file) no longer
|
|
257
|
+
costs memory by the square of its rows: 20,000 rows ran out of memory.
|
|
258
|
+
- Found in a further review, before release:
|
|
259
|
+
- A "not shown" check, and a check that the URL does not hold a text,
|
|
260
|
+
passed on the page a click was about to leave (a redirect a moment
|
|
261
|
+
after the click, to a page that answers slowly). The moment a passing
|
|
262
|
+
state must hold is now timed between two looks, and the URL is read
|
|
263
|
+
once a navigation on its way has arrived.
|
|
264
|
+
- A Set-Cookie pattern is run only on the cookie it names. Written with
|
|
265
|
+
no `^`, or with a second branch (`^id=.*Secure|HttpOnly`), it held on
|
|
266
|
+
another cookie. Exported tests check it the same way.
|
|
267
|
+
- An Outline with several Examples blocks: a `<column>` that only
|
|
268
|
+
another block has is an error. Before, the block without it replayed
|
|
269
|
+
with the text `<column>` as written.
|
|
270
|
+
- "The price differs from before" no longer passes when the element
|
|
271
|
+
stopped showing a price ("Price unavailable"): a price against a text
|
|
272
|
+
that is no amount is refused. Exported tests refuse it the same way.
|
|
273
|
+
- A row value followed by a point or comma at the end of a selector's
|
|
274
|
+
text (`"Qty <n>."`) is kept whole: row 1 no longer finds `Qty 1.5 kg`.
|
|
275
|
+
A minus written as another hyphen (U+2010, U+2011) is read as one.
|
|
276
|
+
- "Not shown" finds text the page writes in the other Unicode form (`é`
|
|
277
|
+
as `e` and an accent).
|
|
278
|
+
- A click whose handler keeps the page busy past the action timeout is
|
|
279
|
+
not clicked again by `retries`.
|
|
280
|
+
- Oracle: text is bound as `CHAR`, so it matches a `CHAR` or `NCHAR`
|
|
281
|
+
column as a literal does. Before, `code = $1` found no row of a
|
|
282
|
+
`CHAR(10)` column, and "not in the database" passed with the row
|
|
283
|
+
there. A `NUMBER` reads as a number whatever its decimals (`12.5`
|
|
284
|
+
read as text, `12` as a number, so no one check served both rows of
|
|
285
|
+
an Outline); only one past 15 digits reads as text. `SELECT UNIQUE`,
|
|
286
|
+
`NEXTVAL` in a check, and the `UTL_` and `DBMS_` packages are refused.
|
|
287
|
+
- Database checks: more aggregates are read as the one row they always
|
|
288
|
+
return (`json_group_array`, `total`, `median`, `jsonb_agg_strict`,
|
|
289
|
+
`corr`, `regr_*`, `range_agg`, `APPROX_COUNT_DISTINCT` and the like),
|
|
290
|
+
so a count or an exists on that row is refused.
|
|
291
|
+
- SQLite: a write to a table declared `ON CONFLICT REPLACE` is refused
|
|
292
|
+
while cleanup is on, as it deletes the row in its way for good. A
|
|
293
|
+
"not in the database" check that sends `true` or `false` as text
|
|
294
|
+
against a boolean is refused, as a number sent as text is.
|
|
295
|
+
- MariaDB: `/*M! ... */` comments, which MariaDB runs, are refused as
|
|
296
|
+
`/*! ... */` is.
|
|
297
|
+
- DynamoDB and Firestore: a "not in the database" check is red on an
|
|
298
|
+
attribute no item has that the filter needs to exist (`$exists`
|
|
299
|
+
true), and on a set or a list compared with a value by equality.
|
|
300
|
+
- MongoDB: a "not in the database" check is red on a field no document
|
|
301
|
+
has in a range, `$exists` true or `$elemMatch` too; a document found
|
|
302
|
+
as the bind is sent is the check's own red, with no type blamed; a
|
|
303
|
+
seed under a number `_id` past int32 no longer fails after writing
|
|
304
|
+
its document; `"$1:uuid"` binds a UUID, which reads as its dashed
|
|
305
|
+
text; and a suite whose MongoDB is down waits 2 seconds for each
|
|
306
|
+
scenario after the first, not 10.
|
|
307
|
+
- A response body's credentials stay hidden when a masked number left
|
|
308
|
+
the body no JSON (`"id": {env:USER_ID}`); before, every token in that
|
|
309
|
+
response stayed in the transcript. A credential repeated inside
|
|
310
|
+
another text (a link, a hint) or in JSON kept as text is hidden too.
|
|
311
|
+
`auth_key`, `authKey` and `X-Auth-Key` are read as credentials, and a
|
|
312
|
+
key in a base URL's query no longer reaches the agent's prompt.
|
|
313
|
+
- A contract's `pattern` that only JavaScript's unicode flag refuses
|
|
314
|
+
(`[\w-\.]`, `\_`) no longer makes every schema check fail to compile,
|
|
315
|
+
and an operation is found by a path the URL encodes (`/price list/{id}`).
|
|
316
|
+
- Masking no longer takes seconds on one long run of letters joined by
|
|
317
|
+
`-` or `.`, and a request with a large body is masked once, not per
|
|
318
|
+
field.
|
|
319
|
+
- A repaired database check's proposal is replayed on later runs and
|
|
320
|
+
on an Outline's other rows. Before, the agent repaired it again on
|
|
321
|
+
every run, and the other rows went red.
|
|
322
|
+
- A pending heal proposal is replayed once the failed replay's own
|
|
323
|
+
database rows are undone, so a `Given` that seeds a row under a
|
|
324
|
+
literal unique value no longer makes it "not replay" and heal again
|
|
325
|
+
on every run.
|
|
326
|
+
- A heal whose proof marks an action as opening a tab files a proposal
|
|
327
|
+
bound to the committed recording as its file holds it. Before, it was
|
|
328
|
+
stale the moment it was filed: `saffron accept` refused it and the
|
|
329
|
+
next run healed again.
|
|
330
|
+
- A heal that changed nothing shows the step the committed recording
|
|
331
|
+
failed on, and names it in the error. Before, every step read as
|
|
332
|
+
passed on a red.
|
|
333
|
+
- A proof of refined steps that cannot open stops the scenario on that
|
|
334
|
+
error. Before, the refined steps were filed with the first proof's
|
|
335
|
+
failure.
|
|
336
|
+
- A browser that dies while a recording is set up is no longer reported
|
|
337
|
+
as the AI provider's fault ("not run: provider").
|
|
338
|
+
- `saffron accept --all` holds more checks recorded again looser: a
|
|
339
|
+
longer time limit, another header, schema or JSON field of a group, a
|
|
340
|
+
group check that gained a filter, another bind or query of a database
|
|
341
|
+
check, and a URL pattern that lost its `^` or `$`.
|
|
342
|
+
- A scenario with no name is an error. It was recorded, then its
|
|
343
|
+
recording and proposal could never be read.
|
|
344
|
+
- A fixture's proposal can apply its feature edits
|
|
345
|
+
(`accept --with-feature-edit`).
|
|
346
|
+
- A URL holding a row's value as the browser writes it
|
|
347
|
+
(`Smith,%20John`) is recorded as its `<column>`, so each row checks
|
|
348
|
+
its own; a recording that kept the recorded row's value there is
|
|
349
|
+
recorded again.
|
|
350
|
+
- The MCP `verify` tool runs one call at a time: two at once could
|
|
351
|
+
answer with each other's scenarios. The MCP and language servers
|
|
352
|
+
follow a `.env` that changed while they run.
|
|
353
|
+
- `run --plan` no longer says a MongoDB, DynamoDB or Firestore insert is
|
|
354
|
+
kept under `db.cleanup: "inserts"`.
|
|
355
|
+
- `saffron coverage --fail-on-loss` (or `--markdown`) with no `--base`,
|
|
356
|
+
and `saffron report --strict` with no `--merge`, exit 2 instead of
|
|
357
|
+
passing over the flag and exiting 0.
|
|
358
|
+
- `saffron init` leaves a `CLAUDE.md` that only reads `AGENTS.md` in
|
|
359
|
+
(`@AGENTS.md`) as it is, where it wrote a second copy of the block,
|
|
360
|
+
and adds to a `.gitignore` in the file's own line endings.
|
|
361
|
+
- Found in a third review, before release:
|
|
362
|
+
- Ctrl-C while the agent records starts no new agent session, ends the
|
|
363
|
+
one running, and files no proposal for the scenario it was on. A run
|
|
364
|
+
stopped by its terminal closing (SIGHUP) stops as Ctrl-C does: it
|
|
365
|
+
writes its report, runs `teardown` and exits 129.
|
|
366
|
+
- `--rerecord` sets each recording aside instead of deleting it, and
|
|
367
|
+
puts it back when the run is killed before recording it again. The
|
|
368
|
+
next run puts back what a killed run left aside, unless a recording
|
|
369
|
+
or proposal for that scenario was written since.
|
|
370
|
+
- A fixture the replay phase ran is not run again when a scenario of
|
|
371
|
+
its feature goes on to the agent.
|
|
372
|
+
- Two runs writing `.saffron/reports/latest.json` at once no longer
|
|
373
|
+
lose each other's scenarios.
|
|
374
|
+
- `saffron import` refuses a bundle whose evidence or proposal would be
|
|
375
|
+
written through a symbolic link out of `.saffron`.
|
|
376
|
+
- Recording: a response field or header checked against the value that
|
|
377
|
+
same response gave (extracted in the When) is refused, as it holds
|
|
378
|
+
whatever the API answered; so is a header checked against a value
|
|
379
|
+
extracted in the same Then. A group's status check names a status
|
|
380
|
+
the step names, as a status check does.
|
|
381
|
+
- Recording: an `{env:...}`, `{data:...}` or kept value that is also a
|
|
382
|
+
column's name is never written inside its `<column>` placeholder,
|
|
383
|
+
where replay typed the literal text `<user>`. A `{data:...}` value
|
|
384
|
+
inside a password no longer leaves the rest of the password in the
|
|
385
|
+
recording. A capture whose name holds an env value is renamed with
|
|
386
|
+
its references, and a request's extract names are renamed too.
|
|
387
|
+
- Recording: a kept value of tens of thousands of characters no longer
|
|
388
|
+
makes every later request fail to record. An action no saved
|
|
389
|
+
recording could load is refused when recorded: a capture named `""`,
|
|
390
|
+
a blank text check (an empty field is checked by its value), a
|
|
391
|
+
status written `" 404"`. A status class written `4XX` is kept as
|
|
392
|
+
`4xx`. A row value a JSON check's pattern holds escaped is its
|
|
393
|
+
column, as in other patterns. A time limit written in words ("within
|
|
394
|
+
two seconds") is the bound it names.
|
|
395
|
+
- Exported tests check as replay does: a pattern or capture never reads
|
|
396
|
+
across a block's edge ("Qty 1" over "0 left" is not "Qty 10"), a
|
|
397
|
+
download holds the value as written, never across a line, "not
|
|
398
|
+
shown" reads words as replay reads them, a role's name in another
|
|
399
|
+
case counts as shown and an option a closed dropdown does not show
|
|
400
|
+
does not, credentials in a response are hidden and a URL's secrets masked as
|
|
401
|
+
replay does, a captured credential encoded in a URL goes to the APIs
|
|
402
|
+
under test only, and a multipart request never sends
|
|
403
|
+
`saffron.config.json`. A check on a target recorded outside an
|
|
404
|
+
iframe is exported as stricter, as the export looks for it in the
|
|
405
|
+
page only. `saffron eject` warns that `setup`, `teardown` and `reset`
|
|
406
|
+
are not run by the exported tests, and a data list the data does not
|
|
407
|
+
hold refuses only the scenario that reads it.
|
|
408
|
+
- Database checks: `count(*)` reads as text on PostgreSQL, MySQL,
|
|
409
|
+
MariaDB and CockroachDB, and as a number on SQL Server, Oracle and
|
|
410
|
+
SQLite, as the guide now says. Oracle refuses more network and
|
|
411
|
+
scheduling packages (`UTL_URL`, `DBMS_LDAP`, `HTTPURITYPE` and the
|
|
412
|
+
like). SQLite's refusal of `true` or `false` sent as text says to
|
|
413
|
+
send 1 or 0.
|
|
414
|
+
- `saffron accept --all` holds a check recorded again without its
|
|
415
|
+
contract check, or on another request group.
|
|
416
|
+
- Found in a fourth review, before release:
|
|
417
|
+
- A replay that failed, then passed when the recording was replayed
|
|
418
|
+
again for a heal, is red with a note that it failed on replay and then
|
|
419
|
+
passed, where it was green. Without an agent it is not replayed again.
|
|
420
|
+
- Database checks: a "not in the database" check is judged again with
|
|
421
|
+
its binds as the other types the database may keep them as, on every
|
|
422
|
+
engine: on MongoDB, DynamoDB and Firestore a range's two ends and the
|
|
423
|
+
values of `$all` together; on PostgreSQL, CockroachDB, MySQL, MariaDB
|
|
424
|
+
and SQLite, text that reads as a number, `true` or `false` as 1 or 0,
|
|
425
|
+
and the values inside a JSON bind. A `count(*)` checked as 0 on SQLite
|
|
426
|
+
and an "at most" count are judged too. Each held with the row there.
|
|
427
|
+
- MongoDB: `$elemMatch` on a field held as one value is refused in a
|
|
428
|
+
"not in the database" check; so are, on DynamoDB and Firestore, a list
|
|
429
|
+
or set compared by `in`, `array-contains`, `$in` or `$beginsWith`, and
|
|
430
|
+
one value compared by `array-contains`. Each held whatever the item
|
|
431
|
+
held.
|
|
432
|
+
- MongoDB, DynamoDB and Firestore take a `:binary` bind (`0x` hex), so
|
|
433
|
+
a field held as bytes can be checked. MongoDB and DynamoDB take a null
|
|
434
|
+
bind, which finds a field held as null, where it was refused and the
|
|
435
|
+
agent was told to write `$exists: false`, which never finds one.
|
|
436
|
+
- DynamoDB: a probe value DynamoDB will not compare (BETWEEN a number
|
|
437
|
+
and a text) no longer fails a valid check. With `db.url` naming a
|
|
438
|
+
region, an endpoint set in the environment (`AWS_ENDPOINT_URL_DYNAMODB`)
|
|
439
|
+
or a profile is no longer used. On DynamoDB and Firestore, a read that
|
|
440
|
+
runs past its time on a database that answers is the check's red,
|
|
441
|
+
never an outage.
|
|
442
|
+
- SQL: in a "not in the database" check, a `LIKE` bind holding a
|
|
443
|
+
backslash (a `[` on SQL Server) is refused, as it never finds the text
|
|
444
|
+
it holds. `$0` is refused. The latest row written as an anti-join, a
|
|
445
|
+
`max()` read through a derived table, and SQL Server's `FOR JSON` and
|
|
446
|
+
`FOR XML` count as keeping one row. On SQL Server, an alias that is a
|
|
447
|
+
word a statement starts with (`SHUTDOWN`) is refused, and a date
|
|
448
|
+
written as text reads year, month, day whatever the login's language.
|
|
449
|
+
Oracle refuses every `UTL_` and `DBMS_` package called (`DBMS_LOB`
|
|
450
|
+
aside) and `APEX_WEB_SERVICE`, as the guide says.
|
|
451
|
+
- MySQL and MariaDB: an `UPDATE` that changed rows its `WHERE`, read
|
|
452
|
+
first, did not find (a `COLLATE` the snapshot lost) is rolled back,
|
|
453
|
+
where cleanup left them changed; a write waiting on a table's lock
|
|
454
|
+
gives up at the step's time, where it waited a year; an `UPDATE` of a
|
|
455
|
+
`FLOAT` under cleanup is refused, as the driver reads it to 6 digits.
|
|
456
|
+
- Schema repairs: a `count(*)`, `LIMIT` or `DISTINCT` check can be
|
|
457
|
+
repaired (each repair was red); a repair of a step's second or later
|
|
458
|
+
check is taken from the repaired check alone, as the agent is told to
|
|
459
|
+
record it; a check that expects no rows, which no repair could be held
|
|
460
|
+
to, starts no agent session.
|
|
461
|
+
- A recording made again (`--rerecord`, or a stale one) whose step now
|
|
462
|
+
writes to the database is held for review against the one it replaces,
|
|
463
|
+
and a first recording whose When writes a table a later Then reads is
|
|
464
|
+
held too. Its note keeps the name `database write added by the heal:`,
|
|
465
|
+
so Accept All in an editor holds it as before. The block `saffron init`
|
|
466
|
+
writes into `AGENTS.md` says so to agents: a `When` acts through the
|
|
467
|
+
app, never the database.
|
|
468
|
+
- Recording: a Then that checked the very field the When had typed into,
|
|
469
|
+
for the value typed and with no page loaded since, passed whether or
|
|
470
|
+
not the app saved it: it is refused, and the agent checks where the
|
|
471
|
+
app shows the saved value, or the field once the page loads again. An
|
|
472
|
+
item or every check that compared a list's entries with a value taken
|
|
473
|
+
from that same list is refused, as the same check on the taken field
|
|
474
|
+
was. A database check bound to a `{unique:...}` no Given or When
|
|
475
|
+
writes, a `{data:...}` no step writes, or a `<table:key>` no table
|
|
476
|
+
holds is refused (a misspelt `{unique:email}` held "not in the
|
|
477
|
+
database" for good).
|
|
478
|
+
- Recording: a time limit in words or in parts is the bound it names:
|
|
479
|
+
"twenty-five seconds" was read as 5 s, "1 minute 30 seconds" as 60 s,
|
|
480
|
+
and "within a second" or "half a minute" as no limit. A response header
|
|
481
|
+
kept as `Header:Location` is kept as `header:Location`, which the
|
|
482
|
+
recording format loads. A group check filtered by a status the step
|
|
483
|
+
does not spell out is recorded: only a status a check expects must be
|
|
484
|
+
named.
|
|
485
|
+
- Recording: a tab closed or opened under a Then was recorded and not
|
|
486
|
+
held, so "the payment window closes" passed when the agent closed it:
|
|
487
|
+
it is held for review, as a click there is. A value of tens of
|
|
488
|
+
thousands of characters (an upload's base64, a long `{data:...}` text)
|
|
489
|
+
made recording fail with "Regular expression too large" after its
|
|
490
|
+
request was sent: it is compared as it stands. Two writes a step asks
|
|
491
|
+
for to one operation were kept as one when they sent a form, a
|
|
492
|
+
multipart, or JSON bodies with other fields (`{sku, quantity}`, then
|
|
493
|
+
`{sku, amount}`): only a field renamed, retyped or added to the same
|
|
494
|
+
values marks a corrected attempt. A capture named with a space whose
|
|
495
|
+
name held an `{env:...}` value had its references garbled: they follow
|
|
496
|
+
its new name. An Outline's database or response field check that
|
|
497
|
+
states a value of the row an earlier step wrote, read through
|
|
498
|
+
`{captured:...}`, follows the row, where every other row was refused.
|
|
499
|
+
- Browser checks: a text check read the items of a flex row, a grid or
|
|
500
|
+
floated boxes a space apart, so "$15" passed on a price drawn as `$15`
|
|
501
|
+
`.99`, and "$50.00" on `-` `$50.00`: a number that goes on across items
|
|
502
|
+
reads as one, and "$12.50" holds on `$` `12` `.50`. Text drawn with no
|
|
503
|
+
area (font-size 0, or in a box scaled to nothing such as a closed
|
|
504
|
+
menu's `transform: scaleY(0)`) counted as shown, so "I see Delete
|
|
505
|
+
account" and "Delete account is not shown" both passed on one page: it
|
|
506
|
+
counts as not shown, and the failure says why. A page rewriting its
|
|
507
|
+
history every few hundred milliseconds kept every settled check from
|
|
508
|
+
ever passing: only a navigation to another document starts the hold
|
|
509
|
+
again. Exported tests read the same.
|
|
510
|
+
- An Outline row value before a point in an XPath function replay
|
|
511
|
+
cannot tie (`contains(translate(., ...), 'qty <n>.')`) is refused, as
|
|
512
|
+
it was without the point, where row 1 passed on "Qty 1.5 kg". An
|
|
513
|
+
Outline whose database or API check names a column but follows nothing
|
|
514
|
+
of it is refused in that check's own words, where it was told to show
|
|
515
|
+
the value on the page.
|
|
516
|
+
- `saffron accept --all` holds a Then recorded again on a wider element
|
|
517
|
+
when only its selectors, its role or its frame changed (a total read on
|
|
518
|
+
`body`, a "not shown" that lost its payment frame). A hand-edited
|
|
519
|
+
recording with an HTTP field on a browser action is refused by name,
|
|
520
|
+
where the staleness check crashed on it.
|
|
521
|
+
- API steps: a request carried an `{env:...}` value or a captured
|
|
522
|
+
credential to another host when it came inside a captured value (a
|
|
523
|
+
`Location` that echoed `?key=` back on another host, captured and
|
|
524
|
+
followed): it is refused, in replay and in exported tests. An exported
|
|
525
|
+
test sent a captured credential written as a query value (`s.urlValue`)
|
|
526
|
+
to another host: it is refused, as replay refuses it. A Set-Cookie
|
|
527
|
+
pattern with an unescaped dot in the cookie's name ran on another
|
|
528
|
+
cookie the dot also matched: the name is read as written. A schema
|
|
529
|
+
check passed on a JSON body served as `text/plain`, a type the
|
|
530
|
+
contract does not give that response: it is red, naming the type.
|
|
531
|
+
- `run --plan` and the MCP `run_plan` tool plan a stale recording with a
|
|
532
|
+
pending proposal as the proposal replay the run makes, where they said
|
|
533
|
+
"record again (AI)". A `baseURL` or `--base-url` that is not an
|
|
534
|
+
`http://` or `https://` URL is refused, naming the key, where every
|
|
535
|
+
API scenario failed with "Invalid URL". `saffron import` refuses to
|
|
536
|
+
write through a `.saffron`, `proposals` or `artifacts` folder that is
|
|
537
|
+
itself a symbolic link.
|
|
538
|
+
- The language server marks a `{unique:...}` that is no token
|
|
539
|
+
(`{unique:e mail}`, `{unique:}`) where it is written, as `saffron run`
|
|
540
|
+
refuses to start for it; JetBrains showed nothing until the run.
|
|
541
|
+
- Found while publishing, and in the review that followed:
|
|
542
|
+
- Playwright colours its error log wherever `TERM`, `COLORTERM`,
|
|
543
|
+
`FORCE_COLOR` or the CI vendor (GitLab CI, CircleCI) give colours, also
|
|
544
|
+
into a pipe or a log file and with `NO_COLOR` set. GitHub Actions and a
|
|
545
|
+
shell without `TERM` get none, which is where this project's tests ran.
|
|
546
|
+
Saffron read that text as it came:
|
|
547
|
+
- The rules that keep an interaction from being sent twice matched
|
|
548
|
+
nothing with the codes in it, so a click whose handler kept the page
|
|
549
|
+
busy past the action timeout was clicked again by the retry.
|
|
550
|
+
- A typed `{env:...}` value that holds a line break (a PEM key, a JSON
|
|
551
|
+
service account) was written in clear to the HTML and JSON reports
|
|
552
|
+
when its fill failed, and was shown to the recording agent, in the
|
|
553
|
+
error's own text where its line breaks stand as typed: the codes
|
|
554
|
+
inside the value kept it from being masked there. A value of 3 to 5
|
|
555
|
+
characters right after a code was left as well.
|
|
556
|
+
- Every failed step's error carried the codes into reports and logs.
|
|
557
|
+
|
|
558
|
+
Playwright's colours are now turned off for the run, and every error is
|
|
559
|
+
read, masked and reported without the codes. The tests run with colours
|
|
560
|
+
on.
|
|
561
|
+
- Known limit, as in 1.4.0: a typed `{env:...}` value of several lines (a
|
|
562
|
+
PEM key, a pasted JSON key file) is masked where its line breaks stand
|
|
563
|
+
as typed, JSON-escaped, percent-encoded or inside base64, and not where
|
|
564
|
+
they are written another way: as single spaces, as a page shows the
|
|
565
|
+
text and the recording agent's snapshot reads it; as CR LF, as a form
|
|
566
|
+
post sends it (`%0D%0A` in a form body); as `↵`, in Playwright's call
|
|
567
|
+
log. There it reaches the agent, the terminal, the reports, the history
|
|
568
|
+
and the trace. Do not type a secret of several lines into a field in a
|
|
569
|
+
scenario. A one-line value with a tab or two spaces in a row is not
|
|
570
|
+
masked where a page shows one space there.
|
|
571
|
+
- A value inside base64 was missed when other text was glued before the
|
|
572
|
+
encoding by `/`, `-` or `_` (`/v/<base64>` in a path, `sess-<base64>`
|
|
573
|
+
in an id), or when the encoding's `+`, `/` or `=` were written `%2B`,
|
|
574
|
+
`%2F` or `%3D` in a query string: the trace kept it and the recording
|
|
575
|
+
agent saw it. Such a run is decoded from after each of those characters
|
|
576
|
+
too. Also in 1.4.0.
|
|
577
|
+
- Outline rows: a check that something is no longer shown, in a step that
|
|
578
|
+
does not name the date column, that held the recorded row's day by its
|
|
579
|
+
month's name without the year ("Nov 20", "20 November", "Fri, Nov 20"
|
|
580
|
+
for a cell `2026-11-20` or `{date+5}`) was bound to no row. Every other
|
|
581
|
+
row then asked whether row 1's day was gone, which holds whatever became
|
|
582
|
+
of its own entry. Such a check binds the recording to its row: the other
|
|
583
|
+
rows are refused, with the step to change. A recording made before this
|
|
584
|
+
is recorded once more on the first row with another day. Also in 1.4.0.
|
|
585
|
+
- A database check that a value "differs" passed on the unchanged value
|
|
586
|
+
when the database reads the column as text (a PostgreSQL `numeric`,
|
|
587
|
+
`bigint` or `count(*)`, a MySQL `DECIMAL` or `BIGINT`): `12.50` was
|
|
588
|
+
never the text `12.5`. Text that spells a number is now compared as the
|
|
589
|
+
number, with no type or with `"type": "number"`; `"type": "string"`
|
|
590
|
+
compares the text. A "differs" check typed as a number or a boolean on a
|
|
591
|
+
value of another type is refused, since it held whatever the row held.
|
|
592
|
+
A text column that holds digits is compared as numbers too (`1.10` is
|
|
593
|
+
"the same" as `1.1`, `5` as `5.0`), except a code written with a leading
|
|
594
|
+
zero on either side (a zip `1234` differs from `01234`): where the text
|
|
595
|
+
itself is what matters, give the check `"type": "string"`. Also in
|
|
596
|
+
1.4.0.
|
|
597
|
+
- An "at most" database check (`count(*) AS n` with `max`) passed with any
|
|
598
|
+
number of rows there when its bind was empty or null (an Outline row
|
|
599
|
+
with an empty cell, an empty capture): the count was 0. It is refused,
|
|
600
|
+
as a "not in the database" check is. To count rows whose column is
|
|
601
|
+
rightly empty, compare its length with the number 0
|
|
602
|
+
(`length(note) = $1`: run on PostgreSQL, MySQL, MariaDB and SQLite; on
|
|
603
|
+
Oracle an empty text is NULL, so write `IS NULL`, beside a column
|
|
604
|
+
compared with a bind). Also in 1.4.0. The
|
|
605
|
+
refusal names the way out of its own engine: it named
|
|
606
|
+
`length(note) = $1` on Oracle too, where an empty text and its length
|
|
607
|
+
are NULL, so the check it led the recording agent to counted no row and
|
|
608
|
+
held whatever the table held. On Oracle it says `IS NULL` beside a
|
|
609
|
+
bound column, on SQL
|
|
610
|
+
Server and MongoDB it names no function they do not have, and a null
|
|
611
|
+
bind is called the "at most" check it is.
|
|
612
|
+
- MySQL and MariaDB: a "not in the database" or "at most" check that
|
|
613
|
+
binds an unquoted `true` where the rows are found by its text (a
|
|
614
|
+
`BOOLEAN` or number column holding false or 0, which MySQL reads the
|
|
615
|
+
text as, or a text column holding 'true') was told only that the
|
|
616
|
+
column holds the step's value as another type. It is told what to
|
|
617
|
+
send: 1 or 0 for a `BOOLEAN` or number column (a `CAST` of the column
|
|
618
|
+
to text holds there too), the quoted text for a text one.
|
|
619
|
+
Such a check on a `BOOLEAN` or number column whose row holds false or
|
|
620
|
+
0 is refused though it is right, where 1.4.0 passed it on MySQL: bind 1
|
|
621
|
+
or 0.
|
|
622
|
+
- `npm install saffron-ai` no longer fails with ERESOLVE in a project
|
|
623
|
+
whose own app depends on another version of a database driver (MongoDB
|
|
624
|
+
6, Firestore 8 or 9, Oracle 5 or 26, a pinned AWS SDK, mssql 11, mysql2
|
|
625
|
+
3.11): the optional drivers accept any version at install, and the
|
|
626
|
+
version is checked where a database step loads the driver. Another
|
|
627
|
+
major version than the tests run is refused with the engine, the major
|
|
628
|
+
needed and the version found ("database steps on MongoDB need mongodb
|
|
629
|
+
7; this project has 6.22.0: npm install mongodb@7"). The messages for a
|
|
630
|
+
missing driver name the version to install.
|
|
631
|
+
- An interaction that reached the page is never sent again by a retry: a
|
|
632
|
+
select, a fill, a hover, a drag, an upload, and a key press (a captured
|
|
633
|
+
credential's own included) whose handler kept the page busy past the
|
|
634
|
+
action timeout ran it twice (a dropdown that submits its form sent it
|
|
635
|
+
twice, a drop was dropped twice). It fails as it is, as a click does. So
|
|
636
|
+
does a drag whose drop target came only after the timeout, the mouse
|
|
637
|
+
being down already. An element that is only covered, disabled, hidden,
|
|
638
|
+
read-only or not there yet is still tried again. A step that passed only
|
|
639
|
+
on such a second try is now red: give the page more time with
|
|
640
|
+
`actionTimeoutMs`.
|
|
641
|
+
- A "not in the database" check, and an "at most" check, no longer pass
|
|
642
|
+
with the row there when the read that judges them (the same query with a
|
|
643
|
+
bind sent as another type) fails for a reason that is not the type: the
|
|
644
|
+
app's lock on a row only that read reaches, a timeout, a deadlock or a
|
|
645
|
+
lost connection. Only the database refusing the value as that type counts
|
|
646
|
+
as "not kept as that type"; a lock is read again and is red if held to
|
|
647
|
+
the end of `db.timeout`, and any other error fails the check. Seen on SQL
|
|
648
|
+
Server with a bytes column and a join.
|
|
649
|
+
- An exported Playwright test printed the `{env:...}` value it typed, the
|
|
650
|
+
element it named or the URL it opened when that action failed (a
|
|
651
|
+
disabled or missing field), in the terminal and the CI log, with the
|
|
652
|
+
line, list, dot, json and junit reporters too. Exported actions on such
|
|
653
|
+
a value, or on a captured credential, now run through the helper's
|
|
654
|
+
`masked`, as the checks do, and name the token. Export again to get it.
|
|
655
|
+
(Also in 1.4.0.)
|
|
656
|
+
- A pattern check of an API response, a header or a database row that
|
|
657
|
+
holds `{unique:<column>}` (which recording writes for an Outline) was
|
|
658
|
+
refused on every row with "A regex quantifier cannot contain a row or
|
|
659
|
+
data token": each row is checked against its own generated value.
|
|
660
|
+
- Known cost of the wait a check makes for its action's answer (new in
|
|
661
|
+
this release, below): a page whose latest action left a request open (a
|
|
662
|
+
long-poll or a stream started on load, with the checks straight after
|
|
663
|
+
the page load) makes each of those checks wait half the action timeout,
|
|
664
|
+
3 seconds at most, and hold for half a second: four checks take about
|
|
665
|
+
12 seconds. In replay the checks after the next interaction do not wait
|
|
666
|
+
for it. In an exported test a count, "not shown", "differs", "URL does
|
|
667
|
+
not contain" or response check on such a page waits that long wherever
|
|
668
|
+
it stands, as in 1.4.0, and an exported test of nine checks that wait
|
|
669
|
+
fails on Playwright Test's 30 second test timeout. An answer slower
|
|
670
|
+
than that wait is not waited
|
|
671
|
+
for by a single check, whatever `actionTimeoutMs` is: write the wait
|
|
672
|
+
into the step ("When I pay and wait for the payment API to return
|
|
673
|
+
402"), then check what the answer shows before what it must leave as it
|
|
674
|
+
was.
|
|
675
|
+
- A key press with no target ends at the action's timeout: on a page whose
|
|
676
|
+
key handler never returns the run hung with no verdict and no report. It
|
|
677
|
+
fails saying the page did not answer, and the key is not pressed again.
|
|
678
|
+
- A run whose output goes away (`saffron run | head`, a pager quit) no
|
|
679
|
+
longer dies with exit 1 and a stack
|
|
680
|
+
trace before its teardown and report: it goes on without output, runs
|
|
681
|
+
`teardown`, writes its report and exits with its own verdict. Known
|
|
682
|
+
limit: when the error stream is gone too (`saffron run 2>&1 | head`), a
|
|
683
|
+
`setup`, `reset` or `teardown` command that prints fails on its own
|
|
684
|
+
write, so a
|
|
685
|
+
`setup` that prints stops the run and `echo done && cleanup` does not
|
|
686
|
+
reach the cleanup. Send such a hook's output to a file.
|
|
687
|
+
- With `--browser webkit`, an app that is not running is counted as
|
|
688
|
+
blocked by the environment, as in Chromium and Firefox. It was counted
|
|
689
|
+
as a failure in the app, and with an agent on, a heal session started
|
|
690
|
+
for every scenario.
|
|
691
|
+
- Database outages said as they are: SQL Server with a certificate the
|
|
692
|
+
machine does not trust names the certificate and
|
|
693
|
+
`?trustServerCertificate=true`, where it said the database did not
|
|
694
|
+
answer; an Oracle host that does not resolve is the database not
|
|
695
|
+
answering; a missing SQLite file under a folder with a bracket in its
|
|
696
|
+
name is the environment; an error the app's own trigger raises that says
|
|
697
|
+
"Connection lost" is the statement failing.
|
|
698
|
+
- `saffron runs`, `import --run`, `report --run` and `trace <run>` no
|
|
699
|
+
longer fail when the shell exports `CLICOLOR_FORCE=1` or `GH_FORCE_TTY`:
|
|
700
|
+
gh and az are run with colours and the forced terminal off.
|
|
701
|
+
- `saffron steps` marks a divergent step `◐`, where it shared `●` with a
|
|
702
|
+
recorded one and differed only by a colour that a pipe, a log file and
|
|
703
|
+
an AI agent's shell never get. The language server's hover does too.
|
|
704
|
+
- `saffron init` reads end of input at its question about the examples
|
|
705
|
+
(Ctrl-D, a closed input) as "no" and goes on, where it ended with exit 0
|
|
706
|
+
and nothing set up; it no longer asks when `CI` is set; it says
|
|
707
|
+
`updated` for a `.gitignore` that was already there.
|
|
708
|
+
- `saffron mcp` typed into a terminal says it is a server for an AI
|
|
709
|
+
assistant and how to add it, and exits 2, where it printed nothing and
|
|
710
|
+
waited. The `--filter` refusal no longer ends with two full stops, and a
|
|
711
|
+
recording's target with nothing to find it by is told
|
|
712
|
+
`fallbackSelectors`, the field's name.
|
|
713
|
+
- Playwright 1.65.1 (from 1.64.0; the range is now `^1.65.1`). Run
|
|
714
|
+
`npx playwright install` after upgrading: it brings Chromium 159,
|
|
715
|
+
Firefox 159 and WebKit 27.2. A project with its own `@playwright/test`
|
|
716
|
+
or `playwright` updates that to the same version first
|
|
717
|
+
(`npm install -D @playwright/test@1.65.1`, or
|
|
718
|
+
`npm install -D playwright@1.65.1` where the project depends on
|
|
719
|
+
`playwright`): beside an older one npm
|
|
720
|
+
installs a second Playwright for Saffron, `npx playwright install`
|
|
721
|
+
installs the browsers of the project's own, and the run goes on
|
|
722
|
+
saying a browser is missing. Its trace no longer says which load state a
|
|
723
|
+
wait waited for, so the trace view names it "Wait for load state".
|
|
724
|
+
Playwright 1.65 itself waits for an element inside an `inert` part of
|
|
725
|
+
the page to stop being inert before acting, leaves text the browser does
|
|
726
|
+
not draw (a closed `<details>`) out of accessible names, and matches
|
|
727
|
+
cookies to paths as RFC 6265 says (`Path=/api` no longer goes to
|
|
728
|
+
`/apiv2`): a recording that relied on the old behaviour turns red. The
|
|
729
|
+
recording agent's browser still names elements as Playwright 1.62 does,
|
|
730
|
+
so a new recording can hold a name with such undrawn text too: its
|
|
731
|
+
action or positive check is red on replay, and a "not shown" check on
|
|
732
|
+
that name passes whatever the page shows. Target such an element by a
|
|
733
|
+
test id or a selector.
|
|
734
|
+
- The release gate (`npm run gate`, which `npm publish` runs): the unit
|
|
735
|
+
tests, every database engine one file at a time (it fails where Docker
|
|
736
|
+
does not answer, where it skipped them all), then both end-to-end
|
|
737
|
+
scripts on the bundle that is published.
|
|
738
|
+
- Found in a fifth review, before release:
|
|
739
|
+
- Database checks: a "not in the database" check held whatever the
|
|
740
|
+
database held when the value was kept as a type its bind was never tried
|
|
741
|
+
as: a JSON string, for a bind the statement reads as JSON
|
|
742
|
+
(`data->'ref' = $1`, a `jsonb` column, MySQL's `CAST($1 AS JSON)`), as
|
|
743
|
+
the text 4471 read as JSON is the number; a JSON boolean, for a MySQL
|
|
744
|
+
bind of 1 or 0, as MySQL compares a JSON true with TRUE and never with
|
|
745
|
+
1; and bytes, on every SQL engine, for the `0x` hex a row shows them as.
|
|
746
|
+
Each is tried now, so such a check is red, saying what found the rows
|
|
747
|
+
and how to compare: as text (`data->>'ref' = $1`), `true` or `false`
|
|
748
|
+
unquoted for a JSON boolean (a bind of "true" was told to send 1 or 0)
|
|
749
|
+
and 1 or 0 for a `BOOLEAN` column, the column's hex on that engine. A
|
|
750
|
+
bind the SQL makes JSON of itself (`to_jsonb(CAST($1 AS text))`,
|
|
751
|
+
`JSON_QUOTE($1)`) is refused in such a check.
|
|
752
|
+
- Database checks: a bind only a negation compares (`<> $2`, `NOT IN`,
|
|
753
|
+
`$ne`, `$nin`, `!=`, `not-in`) made a right "not in the database" check
|
|
754
|
+
red, saying to send that bind as another type, which would have broken
|
|
755
|
+
it: it is left as sent, since sent as another type it finds more rows,
|
|
756
|
+
never fewer. On Firestore a query that finds documents as sent blames no
|
|
757
|
+
type for them. A type probe that met the app's lock on the rows (a busy
|
|
758
|
+
SQLite file) failed the check at once: the check reads again, as its
|
|
759
|
+
read loop does.
|
|
760
|
+
- Database checks: a check that a column is not in a row (`/0/discount`
|
|
761
|
+
exists false) on a SQL engine held whatever the column held once the
|
|
762
|
+
`SELECT` left it out or misspelt it, and failed whatever it held once
|
|
763
|
+
the `SELECT` named it: it is refused, saying to write
|
|
764
|
+
`discount IS NULL`; a field inside a JSON column is still checked so. On
|
|
765
|
+
MongoDB, DynamoDB and Firestore a field checked as not in a document is
|
|
766
|
+
red when no document of the collection holds it (a misspelt name), as a
|
|
767
|
+
filter's field already was.
|
|
768
|
+
- Database checks: a count, or a check that there is no second row, under
|
|
769
|
+
a `WHERE` that keeps one row of each group
|
|
770
|
+
(`id IN (SELECT max(id) FROM orders GROUP BY ref)`, a
|
|
771
|
+
`row_number() OVER (...)` column compared in the `WHERE`) passed on a
|
|
772
|
+
double submit: it is refused, as under `LIMIT`. A JSON number past 2^53
|
|
773
|
+
(an external id in a `json`, `jsonb` or Oracle `JSON` column) read
|
|
774
|
+
rounded, so "differs" passed on the unchanged value and "equals" was red
|
|
775
|
+
on the right one: it reads as its digits, as MySQL reads it, and
|
|
776
|
+
Oracle's cleanup, which put it back rounded, puts the JSON back as it
|
|
777
|
+
was.
|
|
778
|
+
- Database checks: the repair of a step's second database check after a
|
|
779
|
+
schema change was refused before any session when an earlier check of
|
|
780
|
+
the step, which still runs as recorded, expects no rows or was recorded
|
|
781
|
+
without its row count: only the check that failed and the checks the
|
|
782
|
+
repair changes are held to those rules. A recording whose `Then` holds a
|
|
783
|
+
database write (a hand edit, a merge) loaded and ran it, so the `Then`
|
|
784
|
+
checked what it had just written: it is refused, and so is a database
|
|
785
|
+
check under a `Given` or `When`.
|
|
786
|
+
- Database steps, while recording: during a heal the agent could seed the
|
|
787
|
+
database, or send a request that writes, under a step replay had already
|
|
788
|
+
run: the write ran, then was refused, and the extra row made the
|
|
789
|
+
`Then`'s right check fail. Such a write, and one the recording format
|
|
790
|
+
refuses, is refused before it runs. A check bound to a value of more
|
|
791
|
+
than about 32,000 characters that the steps spell out (a doc string's
|
|
792
|
+
JSON document) could not be recorded, the tool answering "Regular
|
|
793
|
+
expression too large": such a value is found as it stands.
|
|
794
|
+
- Outline rows: a literal date cell (`2026-11-20`) that a check showed in
|
|
795
|
+
the page's own format ("Friday, November 20, 2026", "20/11/2026"), an
|
|
796
|
+
amount cell an element showed with other decimals ("Pay $12.50" for
|
|
797
|
+
12.5, in a button's name or a text target), and a data row's value with
|
|
798
|
+
outer spaces (`" Oslo "` in a CSV) that the page showed trimmed were
|
|
799
|
+
found by no rule, so every other row checked row 1's value and a "not
|
|
800
|
+
shown" check passed whatever became of its row: the recording is bound
|
|
801
|
+
to its row. The page's ISO spelling of a cell such as `20.11.2026` no
|
|
802
|
+
longer becomes a date that moves with the calendar.
|
|
803
|
+
- Outline rows: a data table cell of three letters or more, or a
|
|
804
|
+
`{data:...}` value of four or more, was written into unrelated words a
|
|
805
|
+
check held ("Pro" in "Products", "Mark" in "Marketing"): after the cell
|
|
806
|
+
or the file changed, or on another row, a check looked for "Maxducts"
|
|
807
|
+
and a "not shown" check passed whatever the page showed. A value is
|
|
808
|
+
written only where it stands as a value, and an edit whose old value
|
|
809
|
+
stays in such a word records the step once more. A `{data:...}` value
|
|
810
|
+
shown inside angle brackets ("Ann <ann@shop.test>") was kept as recorded
|
|
811
|
+
and read as a placeholder, so the recording stayed fresh when the file
|
|
812
|
+
changed and checked the old address: it is written as its reference, and
|
|
813
|
+
a recording made before is recorded again.
|
|
814
|
+
- Outline rows: a data table value equal to the row's value ("alice"
|
|
815
|
+
seeded by a table, and the row's `<user>`) was written as the table's
|
|
816
|
+
cell in a later check ("Signed in as alice"), so every other row checked
|
|
817
|
+
alice's name: it is the row's `<user>`, or kept as recorded and bound to
|
|
818
|
+
its row where the check lists every row's value. A `{unique:<column>}`
|
|
819
|
+
the agent typed as this row's value was recorded as `{unique:alice}`,
|
|
820
|
+
and every other row was refused: it is recorded as the step writes it,
|
|
821
|
+
and each row types its own generated value. Such a recording is format
|
|
822
|
+
version 23, which older runners refuse: they typed the token as text. A
|
|
823
|
+
capture the agent named after the row's value (`alicebooking`) refused
|
|
824
|
+
every other row once a step naming the column read it: a capture's name
|
|
825
|
+
is no row data. An Outline value of about 33,000 characters or more (a
|
|
826
|
+
maximum-length text from a data file) crashed the recording beside an
|
|
827
|
+
attribute check or a response status: it records.
|
|
828
|
+
- Outline rows: a block beside `Examples: {data:roles}` that lacked one of
|
|
829
|
+
the data file's columns ran `<menu>` as written, on the data rows'
|
|
830
|
+
recording, so a row could pass on another row's data (two data files
|
|
831
|
+
missed it the same way when the placeholder was only in a doc string or
|
|
832
|
+
a table), and a placeholder with a space inside its brackets (`<role >`,
|
|
833
|
+
`< role >`) stayed as written on every row, so the rows replayed each
|
|
834
|
+
other's recording. Both are refused when the file is read, naming the
|
|
835
|
+
placeholder.
|
|
836
|
+
- Recording: a `Then` that read back what a `When` had typed was still
|
|
837
|
+
recorded when it checked the text of a textarea, a rich-text editor or a
|
|
838
|
+
select, named the field another way than the fill did (a selector, a
|
|
839
|
+
test id, a name pattern), or checked an Outline row's value the agent
|
|
840
|
+
had typed as its column: each passed whether or not the app saved
|
|
841
|
+
anything. The recording tool decides on the page which field a check
|
|
842
|
+
reads, and refuses these as it refused the value check. A check against
|
|
843
|
+
a value captured in the same `Then` was recorded when it named the
|
|
844
|
+
captured element another way (`[data-testid=total]` for the `h1`
|
|
845
|
+
captured) or a box holding it, comparing the element with itself: it is
|
|
846
|
+
refused, decided on the page too.
|
|
847
|
+
- Recording: a check found by the text it checks, beside a role (role
|
|
848
|
+
`cell`, text "Ada") or a selector scoped only by tags
|
|
849
|
+
(`table td:has-text('Ada')`), held on any cell showing that text; a
|
|
850
|
+
"starts with" or "ends with" check that let any text come between the
|
|
851
|
+
anchor and the value (`^.*ORD`, `2026.*$`) matched the value anywhere; a
|
|
852
|
+
URL check on what the app's base URL holds (`localhost:3000`, its base
|
|
853
|
+
path) held on every page; and a browser response check or network wait
|
|
854
|
+
passed with a JSON pattern matching any text, a status no step names
|
|
855
|
+
(the 500 the app happened to answer), or a class where the step names
|
|
856
|
+
the code (2xx for "201 Created"). Each is refused, the last as the API
|
|
857
|
+
medium's own checks refuse it.
|
|
858
|
+
- Recording: a password typed as `{env:...}` reached the recording agent
|
|
859
|
+
base64-encoded, in a Basic Authorization header that
|
|
860
|
+
`browser_network_request` shows or from page code that encoded the
|
|
861
|
+
field, and reports and traces kept such a value too, outside a trace's
|
|
862
|
+
request headers: a value inside base64 is masked to its token wherever
|
|
863
|
+
text is masked for display. A heal prompt's live tab line showed a page
|
|
864
|
+
URL holding a credential an API step had captured, while the line below
|
|
865
|
+
masked it: the credential is its `{captured:...}` token there and in the
|
|
866
|
+
completed steps. A first recording, not only a heal, filed the agent's
|
|
867
|
+
rewording of a `Then`, which `accept --all --with-feature-edit` then
|
|
868
|
+
wrote into the feature file: a `Then`'s wording is never filed as a
|
|
869
|
+
feature edit.
|
|
870
|
+
- Browser checks: text inside a box closed to no height or no width with
|
|
871
|
+
overflow hidden (a menu or an accordion closed with `max-height: 0`,
|
|
872
|
+
`height: 0` or `width: 0`) counted as shown in every browser, so "I see
|
|
873
|
+
Delete account" and "Delete account is not shown" both passed on one
|
|
874
|
+
page: it counts as not shown, and so do the fields, buttons and list
|
|
875
|
+
items such a box holds, in replay and exported tests alike, while what
|
|
876
|
+
is positioned against a box outside it still counts, as the browser
|
|
877
|
+
draws it. Text in a zero-sized inline `<svg>` is not shown either. A
|
|
878
|
+
target recorded inside an iframe was read as shown while the iframe was
|
|
879
|
+
hidden, zero-sized, off the page or in such a box, and its "not shown"
|
|
880
|
+
check passed too: it counts only while every iframe it was recorded in
|
|
881
|
+
is shown.
|
|
882
|
+
- Browser checks: a check that something is shown or holds a text, a
|
|
883
|
+
value, an amount, a style or an attribute, or that the URL holds a text,
|
|
884
|
+
was judged at its first look, before the request the `When` sent was
|
|
885
|
+
answered, so "the total is still $50.00", "I am still on the sign-up
|
|
886
|
+
page" and an optimistic "1 item" the server refused all passed: when the
|
|
887
|
+
action before it still has a request out, the check waits for the answer
|
|
888
|
+
(half the action timeout, 3 seconds at most: a slower answer is not
|
|
889
|
+
waited for) and passes only on a state that holds for half a second, as a count or a
|
|
890
|
+
"not shown" check does, in replay and exported tests alike. A check
|
|
891
|
+
whose action sent nothing is judged at once, as before.
|
|
892
|
+
- Browser checks: an attribute check in the absence form ("not disabled"
|
|
893
|
+
as `^(false)?$`) passed on an HTML boolean attribute that was there
|
|
894
|
+
(`disabled`, `checked`, `required`, `hidden`, `open` and the like),
|
|
895
|
+
whose empty value the pattern also takes, so "the Save button is not
|
|
896
|
+
disabled" held on a disabled button: such an attribute is set whenever
|
|
897
|
+
it is there, whatever its text. A style check on a shorthand (`outline`,
|
|
898
|
+
`background`) or a font list, recorded in Chromium, was red in WebKit or
|
|
899
|
+
Firefox on a page that looked the same: it is compared by the longhands
|
|
900
|
+
the value sets. Exported tests read the same.
|
|
901
|
+
- Replay: a page whose own script never yields (an endless loop after a
|
|
902
|
+
click) left every check, and the run, waiting for ever, with no red and
|
|
903
|
+
no report: the check fails at its time limit, saying the page did not
|
|
904
|
+
answer, its own script kept it busy, and the run goes on and writes its
|
|
905
|
+
report. A wait whose target has several ways to be found waited the
|
|
906
|
+
whole action timeout on each in turn before failing (four ways, four
|
|
907
|
+
timeouts): they are polled together under one deadline, the first that
|
|
908
|
+
finds an element deciding.
|
|
909
|
+
- API steps: a status a step named only to rule it out ("the API should
|
|
910
|
+
not answer 500", "a 4xx but not 404") counted as named: the agent was
|
|
911
|
+
told to expect it, and a recording that passed only when the API
|
|
912
|
+
answered 500 was kept. Such a status is no longer named, by a request's
|
|
913
|
+
expected status or by a `Then`'s status check. After copies that
|
|
914
|
+
answered otherwise, or a poll that ran out, the agent's next check was
|
|
915
|
+
verified against that unrecorded response, and the reply said the
|
|
916
|
+
request could not be sent: no check reads it now, the transcript marks
|
|
917
|
+
it as not recorded, and the reply says what it answered. A kept value
|
|
918
|
+
was quoted whole in the tool's reply, past the cap on a tool result (a
|
|
919
|
+
list of 6,000 products made a 418 KB reply): it is cut as a body is.
|
|
920
|
+
- API steps: a capture the scenario sent as its credential (an API key
|
|
921
|
+
kept as `key` and sent as `X-Api-Key`, a `Bearer {captured:result}`) was
|
|
922
|
+
a credential only when its name said so: the key reached the transcript,
|
|
923
|
+
the report, error text and the agent, and was sent to another host. A
|
|
924
|
+
capture sent in a credential header, or in a query parameter, form field
|
|
925
|
+
or body field a credential's name names, is a credential whatever its
|
|
926
|
+
name, from the moment it is extracted, in a browser scenario's API
|
|
927
|
+
steps too, where it became one only when a request first sent it, so a
|
|
928
|
+
check failing before then quoted it. An exported test keeps it as one
|
|
929
|
+
too: masked, sent only to the APIs under test, and with the trace left
|
|
930
|
+
out of the generated config.
|
|
931
|
+
- API steps: a schema check ignored a discriminator, so a valid response
|
|
932
|
+
of a `oneOf` picked by `discriminator` (the OpenAPI specification's own
|
|
933
|
+
Pet example) was always red, and such a request body could not be
|
|
934
|
+
recorded: the discriminator's property picks the schema, by its mapping
|
|
935
|
+
or the component's name, in replay and exported tests alike, and a value
|
|
936
|
+
neither names is red. A schema check was red on a response the contract
|
|
937
|
+
allows when its media type had no JSON schema (`"application/json": {}`,
|
|
938
|
+
or a `text/plain` response the contract gives a text schema): a media
|
|
939
|
+
type with no schema leaves the body free (a JSON one must still be
|
|
940
|
+
JSON), and a non-JSON type the contract gives is checked as text against
|
|
941
|
+
its schema. Behind `transport.api.proxy` the API was probed directly
|
|
942
|
+
before an agent session started, so an API whose host resolves only
|
|
943
|
+
through the proxy was never recorded ("did not answer (ENOTFOUND)"): the
|
|
944
|
+
probe goes to the proxy, as every request does, and names a proxy that
|
|
945
|
+
is down.
|
|
946
|
+
- Exported tests: an exported `@api` scenario sent its requests through
|
|
947
|
+
the test's browser context, which Playwright starts with the project's
|
|
948
|
+
`storageState`, so a request whose own token failed passed on the
|
|
949
|
+
signed-in session's cookie, where replay is red: each session of an
|
|
950
|
+
exported `@api` scenario sends through a request client of its own with
|
|
951
|
+
no cookies, as replay does, and a browser action in an `@api` scenario,
|
|
952
|
+
which replay fails, is exported as `test.fixme`. A failed exported
|
|
953
|
+
element check printed the resolved `{env:...}` value it waited for (a
|
|
954
|
+
text, a field's value, a URL, a download's text, or a target named by
|
|
955
|
+
one) in the terminal, the CI log and Playwright's reports: it names the
|
|
956
|
+
`{env:...}` token, as replay's errors do. `saffron eject` warned only of
|
|
957
|
+
a Playwright trace, and said turning it off was enough, while
|
|
958
|
+
Playwright's HTML report keeps every typed `{env:...}` value as a step's
|
|
959
|
+
title: it warns of an HTML or blob reporter in the project's own
|
|
960
|
+
Playwright config too, naming `playwright-report` and `blob-report`
|
|
961
|
+
beside `test-results`.
|
|
962
|
+
- Runs: a heal that recorded the same actions as the committed recording,
|
|
963
|
+
differing only in the page fingerprint a browser recording stamps, was
|
|
964
|
+
filed as a proposal, so a failing step turned yellow: it is red, as any
|
|
965
|
+
heal that changed nothing is. A scenario with no steps of its own (its
|
|
966
|
+
lines read as its description) ran its Background alone and could pass:
|
|
967
|
+
it is red, as a scenario with no steps is. With the app down, a pending
|
|
968
|
+
proposal's replay was blamed on the recording (for another Examples row,
|
|
969
|
+
with advice to reject it and record again) or, under `--no-agent`, on
|
|
970
|
+
the recorder: it is blocked by the environment, as a committed
|
|
971
|
+
recording's replay is. A scenario that needed recording in a browser
|
|
972
|
+
that only replays (Firefox, WebKit) was counted as blocked by the
|
|
973
|
+
agent's provider: it is blocked by the recorder. With no agent and
|
|
974
|
+
several workers, checking a recording again against a changed API
|
|
975
|
+
contract ran the `reset` commands while other workers replayed: it runs
|
|
976
|
+
none, as an ordinary replay.
|
|
977
|
+
- Review: a held note whose own words held a value the scenario names with
|
|
978
|
+
`{env:...}` (`stable` in `unstable recording:`, `base` in
|
|
979
|
+
`database write added`) was masked there too, and `accept --all` then
|
|
980
|
+
promoted the proposal: a note's prefix is never masked. A heal that
|
|
981
|
+
changed or dropped the status a request expects (201 to 409, or to any
|
|
982
|
+
2xx) kept that request's requirement and was not held for review: the
|
|
983
|
+
expected status is part of it. An import that stopped part way (a write
|
|
984
|
+
that failed, Ctrl-C) left the proposals it had already written out of
|
|
985
|
+
the import ledger, and `accept --all` promoted them as if filed here:
|
|
986
|
+
each is in the ledger before it is written. `saffron run --rerecord`
|
|
987
|
+
lost track of the recording it replaced when that one was kept under its
|
|
988
|
+
0.9.9 file name (a scenario name with no ASCII letters, or a long name
|
|
989
|
+
saved whole), so a looser `Then` recorded again was not held: it is
|
|
990
|
+
found. A `--rerecord` run that recorded nothing for a scenario deleted
|
|
991
|
+
the recording it had set aside when the scenario's pending proposal was
|
|
992
|
+
rejected or replaced during the run: it is put back.
|
|
993
|
+
- Configuration: a count or limit written in quotes
|
|
994
|
+
(`"stabilityProofs": "2"`) was read as a number by `saffron run` and as
|
|
995
|
+
text by the MCP `run_plan`, which planned 1121 executions: every command
|
|
996
|
+
refuses it as the config loads, naming the key. A bad value in
|
|
997
|
+
`saffron.config.json` was refused as a flag the user never passed
|
|
998
|
+
(`invalid --trace "yes"`): the message names the key
|
|
999
|
+
(`invalid trace "yes" in saffron.config.json`), and the flag only when
|
|
1000
|
+
one was given. A config saved with a byte order mark (Windows Notepad,
|
|
1001
|
+
PowerShell 5.1) was refused as not valid JSON, and `saffron init` filled
|
|
1002
|
+
`features/` instead of the configured folder: the mark is read as no
|
|
1003
|
+
part of the JSON, as in a data file. `saffron mcp` did not start while
|
|
1004
|
+
`saffron.config.json` was broken, so the agent fixing it had no tools:
|
|
1005
|
+
the server starts, and each tool answers on the defaults with
|
|
1006
|
+
`configProblem` saying what is wrong.
|
|
1007
|
+
- `saffron init` replaced a `saffron` MCP server you had set up yourself
|
|
1008
|
+
(a local CLI, an `env`) with its `npx` entry and said only "updated": it
|
|
1009
|
+
keeps one that runs another command and says so, keeps any field you
|
|
1010
|
+
added to its own entry, and brings an earlier init's `npx saffron mcp`
|
|
1011
|
+
up to date. `saffron report --merge` of a named run's shards wrote
|
|
1012
|
+
`latest.json` and `history.jsonl` without the lock a run takes, so a run
|
|
1013
|
+
writing at the same time could lose its rows, or the merge its own: both
|
|
1014
|
+
write under one lock. `saffron status`, the editors' panels and the MCP
|
|
1015
|
+
`project_status` crashed on a last report whose row held steps or
|
|
1016
|
+
evidence that were not a list, or an error that was not text, and on one
|
|
1017
|
+
with no totals: such a row is read without those fields, and such a
|
|
1018
|
+
report as none.
|
|
1019
|
+
- Parser: a mistyped header (`scenario:`, `Background :`, or
|
|
1020
|
+
`Scenario Name` and `Background` with no colon) over a single step, or
|
|
1021
|
+
over `*` steps, was read as description and silently did not run, and a
|
|
1022
|
+
file of one such scenario ran nothing: it is refused, naming the header.
|
|
1023
|
+
A `*` line counts as a step only right under a header misspelt with its
|
|
1024
|
+
colon, and one step under a header word with no colon only when it is
|
|
1025
|
+
indented deeper, so bullets and prose in a description stay description.
|
|
1026
|
+
A StepSet call written after a step keyword (`When StepSet Login`,
|
|
1027
|
+
`* StepSet Login`) ran as an ordinary step the agent made sense of, and
|
|
1028
|
+
the set never ran: it is refused, saying a call is a line of its own. A
|
|
1029
|
+
StepSet at the top of a `.feature` file was told to fix its header's
|
|
1030
|
+
spelling, or with one step under it ran nothing: it is told to rename
|
|
1031
|
+
the file to `.saffron`. A refusal of a line read as description names
|
|
1032
|
+
that line, where it named its block's header (the language server marks
|
|
1033
|
+
it there too), and Gherkin's parse errors quote the line as written, at
|
|
1034
|
+
file:line:column, where they quoted it as Saffron rewrites it
|
|
1035
|
+
(`* StepSet Login`) and lost the file:line form in a file that defines a
|
|
1036
|
+
StepSet.
|
|
1037
|
+
- The language server, which JetBrains reads: one StepSet problem anywhere
|
|
1038
|
+
(a nested call, a name defined twice, a set with Examples, a set file
|
|
1039
|
+
that does not parse) left it no step sets, so every call in every file
|
|
1040
|
+
was marked "not defined anywhere", completion offered nothing, and the
|
|
1041
|
+
problem the runner stops on was marked nowhere: each problem is marked
|
|
1042
|
+
on its own line, and every other set is kept. A `.saffron` description
|
|
1043
|
+
line starting with "StepSet " is no longer marked as a call. Two
|
|
1044
|
+
scenarios of one name in a file (one per Rule), or names that slug
|
|
1045
|
+
alike, which every run of the file refuses as sharing a recording, are
|
|
1046
|
+
marked on the later one. A step with a long run of spaces inside it took
|
|
1047
|
+
more than a second per keystroke: the pattern sessions are read with is
|
|
1048
|
+
linear.
|
|
1049
|
+
- Exported tests: a check of a target inside an iframe passed while the
|
|
1050
|
+
iframe was hidden, zero-sized or inside a box clipped to nothing, where
|
|
1051
|
+
replay fails it: the exported test reads such a target as not shown. On
|
|
1052
|
+
a page that holds a request open (a long-poll, a notifications stream)
|
|
1053
|
+
an exported test waited up to 3 seconds before each check and failed a
|
|
1054
|
+
message shown briefly that replay passed: every action now starts its
|
|
1055
|
+
own window, as in replay.
|
|
1056
|
+
- A DynamoDB "not in the database" check filtering on a top-level
|
|
1057
|
+
attribute whose name holds a dot (`meta.tier`) was refused as naming an
|
|
1058
|
+
attribute no item has: it is read as the attribute it names.
|
|
1059
|
+
- Recordings and files that may turn red or stop a run with this release:
|
|
1060
|
+
a feature file with a mistyped scenario header (fix the header), a
|
|
1061
|
+
database check that counts rows under a one-row subquery (re-record,
|
|
1062
|
+
`saffron run --rerecord`), a "not shown" check on a whole number shown
|
|
1063
|
+
with zero decimals, an amount check on a value shown with three or more
|
|
1064
|
+
decimals that was a little off, a config with `actionTimeoutMs: 0`, an
|
|
1065
|
+
Outline that uses a `<column>` one of its Examples blocks lacks (add the
|
|
1066
|
+
column), a scenario with no name (name it), a Set-Cookie pattern written
|
|
1067
|
+
with a second branch, a "differs" check between a price and a text
|
|
1068
|
+
that is no amount (capture the amount alone), exported tests, once
|
|
1069
|
+
ejected again, that passed on text across a block's edge or a download's
|
|
1070
|
+
lines, a "not in the database" check whose bind is of another type than
|
|
1071
|
+
the database keeps the value as (give the bind its type), a `LIKE` bind
|
|
1072
|
+
with a backslash in such a check, a `$0` placeholder, a SQL Server alias
|
|
1073
|
+
that is a statement's word, an Oracle check that calls a `UTL_` or
|
|
1074
|
+
`DBMS_` package, a count of rows under a latest-row anti-join, a
|
|
1075
|
+
MySQL `UPDATE` of a `FLOAT` under cleanup (set `db.cleanup` to
|
|
1076
|
+
`"inserts"`), a config whose `baseURL` is not an `http://` or
|
|
1077
|
+
`https://` URL, a schema check on a response served as a type that is
|
|
1078
|
+
not JSON, a mistyped header over a single step or `*` steps, or a
|
|
1079
|
+
description line starting with Scenario, Example or Background over a
|
|
1080
|
+
step indented deeper (fix the header, or reword or unindent the line), a
|
|
1081
|
+
StepSet call after a step keyword (drop the keyword), a placeholder with
|
|
1082
|
+
a space inside its brackets (remove the space), an Examples block beside
|
|
1083
|
+
a `{data:...}` one that lacks one of its columns (add the column), a
|
|
1084
|
+
scenario with no steps of its own, only the Background's (give it steps,
|
|
1085
|
+
or remove it), a count or limit in quotes in `saffron.config.json`
|
|
1086
|
+
(`"stabilityProofs": "2"`: write the number), a recording whose `Then`
|
|
1087
|
+
writes to the database or whose `Given` or `When` checks it (re-record),
|
|
1088
|
+
a SQL check that a column is not in a row (re-record, and it is written
|
|
1089
|
+
`IS NULL`), a "not in the database" check on a field no document of the
|
|
1090
|
+
collection holds (fix a misspelt name, or let a document hold the field
|
|
1091
|
+
first), one that compares a JSON value with a bind read as JSON, a MySQL
|
|
1092
|
+
JSON boolean with 1 or 0, or a binary column with `0x` hex, or that
|
|
1093
|
+
binds JSON the SQL makes itself (compare as its message says), a count
|
|
1094
|
+
or a "no second row" check under a `WHERE` that keeps one row of each
|
|
1095
|
+
group (re-record), a check of a state shown only while the `When`'s
|
|
1096
|
+
request is out, such as a spinner (re-record, so it is watched for from
|
|
1097
|
+
the `When`), a check of text in a box closed to no height or width, or
|
|
1098
|
+
in an iframe the page hides (open it in a `When` first), an absence
|
|
1099
|
+
check on an HTML boolean attribute that is there with a text such as
|
|
1100
|
+
`false` (the browser reads it as set: fix the page), an Outline whose
|
|
1101
|
+
check shows a row's date, amount or spaced value in the page's own form
|
|
1102
|
+
(give the shown value an Examples column of its own), and exported
|
|
1103
|
+
`@api` tests, once ejected again, that passed on the signed-in session's
|
|
1104
|
+
cookie (give the request credentials of its own). From the last review:
|
|
1105
|
+
an Outline whose not-shown check holds the recorded row's day by its
|
|
1106
|
+
month's name without the year (the other rows are refused, and a run with
|
|
1107
|
+
no agent calls an older such recording stale: name the date column in the
|
|
1108
|
+
step, or re-record); a database "differs" check on text that spells a
|
|
1109
|
+
number (`"type": "string"` compares the text); a database "at most" check
|
|
1110
|
+
with an empty or null value (give the step a value, or count empty
|
|
1111
|
+
columns by `length(note) = $1` with 0; on Oracle an empty text is NULL,
|
|
1112
|
+
so write `IS NULL` beside a column compared with a bind); database steps with a driver
|
|
1113
|
+
of another major version than the tests run (the refusal names the one
|
|
1114
|
+
to install); a step whose select, fill, hover, drag, upload, check or
|
|
1115
|
+
key press passed only on its second try, after its handler kept the page
|
|
1116
|
+
busy past the action timeout (it is no longer sent twice: raise
|
|
1117
|
+
`actionTimeoutMs`); a click, a key press, a check, a drag or a hover that closes
|
|
1118
|
+
its own tab and is rejected by Playwright for it, where a second try used to
|
|
1119
|
+
land on the tab behind (the step is red at once); a recording that
|
|
1120
|
+
relied on Playwright 1.64 acting inside an `inert` part of the page, on
|
|
1121
|
+
an accessible name that holds text the browser does not draw, or on a
|
|
1122
|
+
cookie whose `Path` is only a prefix of the request's (re-record); a
|
|
1123
|
+
database "differs" check typed as a number or a boolean on a value the
|
|
1124
|
+
row holds as another type (give it the type the row shows, or none); a
|
|
1125
|
+
MySQL or MariaDB "not in the database" or "at most" check that binds an
|
|
1126
|
+
unquoted `true` on a `BOOLEAN` or number column holding false or 0 (bind
|
|
1127
|
+
1 or 0); an
|
|
1128
|
+
Outline whose not-shown check holds a word that is also a month's name
|
|
1129
|
+
("may", "march") beside a number that is a row's day (the other rows are
|
|
1130
|
+
refused: name the date column in the step); and exported tests, once
|
|
1131
|
+
ejected again, whose nine or more checks straight after a page load that
|
|
1132
|
+
keeps a request open now each wait about 3 seconds (raise the test's
|
|
1133
|
+
`timeout` in `playwright.config.ts`).
|
|
1134
|
+
|
|
1135
|
+
Editors (VS Code extension 0.8.3, JetBrains plugin 0.8.3)
|
|
1136
|
+
|
|
1137
|
+
- VS Code: the same headers and steps Gherkin reads as description are
|
|
1138
|
+
marked as errors; a StepSet the file itself defines is no longer called
|
|
1139
|
+
undefined, and a description line starting with "StepSet " is text; a
|
|
1140
|
+
`Background:` line under a scenario's header no longer hides its
|
|
1141
|
+
Examples errors; a `.feature` file in a folder with no
|
|
1142
|
+
`saffron.config.json` gets no Saffron errors; a row of a report written
|
|
1143
|
+
on Windows opens its scenario; with no folder open the panel says so
|
|
1144
|
+
instead of loading for ever; `{data:...}` hints read a data file once
|
|
1145
|
+
until it changes. More of what the runner's parser refuses is marked
|
|
1146
|
+
where it is written: a scenario with no name, an outline placeholder
|
|
1147
|
+
no Examples row fills (a column only another Examples block has, a
|
|
1148
|
+
near miss of a column, a column named twice, placeholders with no
|
|
1149
|
+
Examples, a column a data file lacks), a `{unique:...}` that is no
|
|
1150
|
+
token, a StepSet line anywhere in a `.feature` file, and a fixture or
|
|
1151
|
+
step set the runner cannot run.
|
|
1152
|
+
- JetBrains: the CI Runs results page runs none of the scripts a run's
|
|
1153
|
+
bundle carries; a row of a report written on Windows opens its scenario
|
|
1154
|
+
at its line; a feature file that is not valid UTF-8 is listed with its
|
|
1155
|
+
scenarios; Run Selected on one file names its configuration by the
|
|
1156
|
+
file's path; a test folder outside the project is watched; a selected
|
|
1157
|
+
scenario row in CI Runs stays selected through a reload, told apart
|
|
1158
|
+
from another feature's of the same name. A folder of feature files
|
|
1159
|
+
added to the test folder is listed without a Refresh, and on Windows a
|
|
1160
|
+
test folder on another drive is listed at all; a status reload keeps
|
|
1161
|
+
the proposal being read and the Last Run row selected; while the status
|
|
1162
|
+
loads, the tabs say so instead of claiming nothing is pending; the
|
|
1163
|
+
Accept All tooltips name a database write a new recording added.
|
|
1164
|
+
|
|
8
1165
|
## 1.4.0: Database steps
|
|
9
1166
|
|
|
10
1167
|
Released 2026-11-10.
|
|
@@ -593,7 +1750,7 @@ Runner
|
|
|
593
1750
|
row, with guidance, where another row passed on it.
|
|
594
1751
|
- The recording agent cannot check a value against one captured in the same
|
|
595
1752
|
Then, nor a response by a pattern every URL matches, nor bind a database
|
|
596
|
-
check to part of a row's value ("ann" of ann@example.test). A watch armed
|
|
1753
|
+
check to part of a row's value ("ann" of `ann@example.test`). A watch armed
|
|
597
1754
|
in a step replay ran before the session can be checked.
|
|
598
1755
|
- A recording the post-process refuses is told to the agent and ends as an
|
|
599
1756
|
ordinary red, its cost counted, not as the provider failing.
|
|
@@ -3032,7 +4189,7 @@ Runner
|
|
|
3032
4189
|
("red" in "Delivered", "Pro" in "Products") no longer binds the row; a
|
|
3033
4190
|
camelCase part ("SelectedRed") still does. Rows whose data reads blank
|
|
3034
4191
|
on every row no longer record each other forever. A row value holding a
|
|
3035
|
-
secret's value (
|
|
4192
|
+
secret's value (`admin@shop.test` beside a variable "admin") is the
|
|
3036
4193
|
row's `<column>`, and a secret equal to a row's lookup key leaves the
|
|
3037
4194
|
step's `{data:...}` reference alone. A state check ("Promo" hidden, an
|
|
3038
4195
|
attribute's pattern) binds the row whatever its element is named. A
|