saffron-ai 1.3.0 → 1.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +788 -0
- package/dist-pkg/cli.js +396 -376
- package/package.json +1 -1
- package/skills/saffron/SKILL.md +13 -2
- package/skills/saffron/references/config.md +3 -2
- package/skills/saffron/references/syntax.md +9 -0
- package/templates/playwright/saffron-helpers.ts +399 -74
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,794 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.3.2: Exact checks for amounts, URLs and numbers, and secrets kept masked
|
|
9
|
+
|
|
10
|
+
Released 2026-11-04.
|
|
11
|
+
|
|
12
|
+
Runner
|
|
13
|
+
|
|
14
|
+
Upgrading from 1.3.1:
|
|
15
|
+
|
|
16
|
+
- A URL check (`expectUrl`, contains) reads its text as a whole part of
|
|
17
|
+
the URL, and a path (text with a `/`) outside the query, unless the text
|
|
18
|
+
names a query (`?`, `=`, `&`): `/dashboard` no longer holds on
|
|
19
|
+
`/login?next=%2Fdashboard`, nor `/orders/42` on `/orders/421`. A word
|
|
20
|
+
with no slash (`shoes`) is still found in `/search?q=red+shoes`, but not
|
|
21
|
+
in a query value that is a path (`dashboard` in `/login?next=/dashboard`),
|
|
22
|
+
nor in a query value whose encoded `#` or `&` makes it look like a
|
|
23
|
+
fragment or another field (`/dashboard` in `?next=%2F%23%2Fdashboard`).
|
|
24
|
+
A check that the URL does not hold a text still reads it anywhere:
|
|
25
|
+
`error` is in `?errorCode=5`. A text that names a query is the page's
|
|
26
|
+
own: its `?` is where the URL's query starts and a field it names is
|
|
27
|
+
one of the URL's own, so `/dashboard?tab=settings` and `tab=settings`
|
|
28
|
+
no longer hold on `/login?next=%2Fdashboard%3Ftab%3Dsettings`, nor
|
|
29
|
+
`/dashboard` on an OAuth callback's `#state=%2Fdashboard`. A fragment's
|
|
30
|
+
fields are the page's own too: `#tab=settings`, `/app#tab=settings` and
|
|
31
|
+
`tab=settings` hold on `/app#tab=settings`. A recording
|
|
32
|
+
that passed only that way turns red; the exported test checks the same.
|
|
33
|
+
- `Examples: {data: roles}` (a space, a capital `Data`) is an error naming
|
|
34
|
+
the file and line, where the Outline used to run no row at all. The
|
|
35
|
+
language server marks it while it is typed.
|
|
36
|
+
- An expected text never matches inside a longer word or number, whether
|
|
37
|
+
a row value or written as it is (SAF-197): `"$10"` no longer holds on
|
|
38
|
+
`$100`, `"1 item"` on `11 items`, nor a text target `Saved` on "You have
|
|
39
|
+
unsaved changes". A role's name may still have grown a suffix. A number
|
|
40
|
+
written in the step may be followed by a zero decimal part (`"$10"` in
|
|
41
|
+
`$10.00`, never `$10.50`, `₹1,00,000` or `2.0.1`); a row value keeps its
|
|
42
|
+
decimals whole (`<id>` 1 is not in `1.00`), and neither is found before
|
|
43
|
+
a digit in an inline tag (`Order 1<b>2</b>`, `$10<sup>.50</sup>`). A
|
|
44
|
+
text ending in a number's point or comma takes no more digits
|
|
45
|
+
(`"Refund of $10."` is not in `Refund of $10.50`), and a number is not
|
|
46
|
+
found in its negative (`"5.00"` is not in `-5.00` or `−5.00`; `"5"` is
|
|
47
|
+
still in `pages 3-5`). A value that starts or ends in Chinese, Japanese,
|
|
48
|
+
Thai, Lao, Khmer or Myanmar script is not bounded there, as those
|
|
49
|
+
scripts put no space between words, but for a negating prefix:
|
|
50
|
+
`"成功"` is not in `不成功` nor `没有成功`, `"保存"` not in `无法保存`.
|
|
51
|
+
Only a negation right before the value is read: a short value in a
|
|
52
|
+
sentence that negates it further back can still be found. A recording that passed only that way turns red, and the agent
|
|
53
|
+
records the whole word next time.
|
|
54
|
+
- An expected text never runs across a block, a table cell or a `<br>`
|
|
55
|
+
(SAF-198): `"15.00"` no longer holds on the cells `1` and `5.00`, nor
|
|
56
|
+
`"Qty 10"` on the items `Qty 1` and `0 left in stock`. Items of a flex
|
|
57
|
+
row or a grid read as one line, a space apart: `"Total: $10"` holds on
|
|
58
|
+
the items `Total:` and `$10`. A value written on several lines still
|
|
59
|
+
matches the lines the page draws.
|
|
60
|
+
- A style or attribute check needs its element shown (SAF-199): "the
|
|
61
|
+
error is shown in red" no longer holds on an error that is not on
|
|
62
|
+
screen, nor on a shown fallback when the recorded element is on the page
|
|
63
|
+
but hidden. A check of what hides an element (`display`, `visibility`,
|
|
64
|
+
`opacity`, `content-visibility`; `hidden`, `aria-hidden`, `inert`) still
|
|
65
|
+
holds on a hidden one, and so does one of a `class`, a `style`, `open`
|
|
66
|
+
or `data-state` attribute, or a height, width, transform or clip: how
|
|
67
|
+
many apps hide an element (`d-none`, a closed dialog).
|
|
68
|
+
|
|
69
|
+
The exported Playwright tests check text, blocks and styles the same way
|
|
70
|
+
(SAF-197, SAF-198, SAF-199).
|
|
71
|
+
|
|
72
|
+
- A text check of an amount written with its currency (`$5.00`, `€5`)
|
|
73
|
+
no longer holds on the negative amount (`-$5.00`, `−€5`), as `5.00`
|
|
74
|
+
already did not on `-5.00`. A URL check of a number takes no decimal
|
|
75
|
+
part (`amount=10` is not in `?amount=10.50`, nor `/orders/42` in
|
|
76
|
+
`/orders/42.5`), and a letter or digit the URL writes percent-encoded
|
|
77
|
+
belongs to the word beside it (`name=Jos` is not in `?name=Jos%C3%A9`,
|
|
78
|
+
nor `id=42` in `?id=42%35`). The exported test checks the same.
|
|
79
|
+
|
|
80
|
+
Fixed, from a review of the runner:
|
|
81
|
+
|
|
82
|
+
- A credential stays masked wherever it travels: a fixture's token that a
|
|
83
|
+
later capture replaced, or one inside an extracted object, in the
|
|
84
|
+
scenarios the fixture seeds; and a replaced token in what the agent
|
|
85
|
+
reads while recording, which it used to show. An exported test masks a
|
|
86
|
+
secret as a URL encodes it (`%2B`, `%2F`, `%3D`) and in the errors of
|
|
87
|
+
its JSON checks, as replay does.
|
|
88
|
+
- A `differs` check on an object holding an integer beyond JavaScript's
|
|
89
|
+
precision no longer passes against the same object a capture kept.
|
|
90
|
+
- The dashboard's suite health no longer reads 100% while a scenario
|
|
91
|
+
failed (473 of 475 reads 99.58%).
|
|
92
|
+
- The language server no longer marks `Examples: {data: roles}` as an
|
|
93
|
+
error inside a doc string, nor when a description comes before its
|
|
94
|
+
table: the runner reads both.
|
|
95
|
+
|
|
96
|
+
- No false green from a fallback selector or an iframe: the first locator
|
|
97
|
+
that finds a shown element decides a check, so a fallback that happens
|
|
98
|
+
to show the expected text no longer vouches for the recorded element
|
|
99
|
+
that failed it. A recorded element that is gone, or hidden for a check
|
|
100
|
+
other than a style or attribute check, still gives way to its fallback.
|
|
101
|
+
- No false green from an API check whose pattern matches any text: a
|
|
102
|
+
field, header or `until` pattern such as `.*`, `^` or an empty one is
|
|
103
|
+
refused while recording, as page checks already were.
|
|
104
|
+
- No Outline row passes on another row's data when the app writes a row
|
|
105
|
+
value without its accents (`José` as `status-jose`), or when a column
|
|
106
|
+
names a section without quotes (`the <plan> card shows "Popular"`).
|
|
107
|
+
- A captured id of one or two characters is recorded as its token where
|
|
108
|
+
a path segment or field names its capture (`/orders/42` for `order`), so
|
|
109
|
+
replay no longer reads the record made while recording. A field that
|
|
110
|
+
names another id (`productId: 1`) is left as it is.
|
|
111
|
+
- A secret in an API query that holds `/` together with `+`, `=` or `&`
|
|
112
|
+
(a base64 key) is masked in errors, transcripts, reports and the
|
|
113
|
+
agent's replies.
|
|
114
|
+
- A page or API message cut for a report is cut with `…`, so a secret
|
|
115
|
+
across the cut is masked too, also when the message carries it
|
|
116
|
+
URL-encoded.
|
|
117
|
+
- A request method taken from a column (`a <verb> request`) is kept as
|
|
118
|
+
sent, which binds the recording to its row, instead of an invalid
|
|
119
|
+
recording made again on every run.
|
|
120
|
+
- An integer above 2^53 extracted from a response (a snowflake id) keeps
|
|
121
|
+
the digits the API sent, instead of being rounded to another id. A JSON
|
|
122
|
+
check reads the same digits, so "no longer the same id" fails on the
|
|
123
|
+
same id, and a request naming the captured id still sends its token.
|
|
124
|
+
- A differs check finds the element again when the app draws it anew
|
|
125
|
+
(a React remount), where it used to fail for good.
|
|
126
|
+
- A shard run removes its own earlier report as it starts, so a shard
|
|
127
|
+
that then fails setup, crashes or is stopped is missing from
|
|
128
|
+
`report --merge` instead of standing in with the earlier run's results.
|
|
129
|
+
It also goes when the run stops before running anything (a feature file
|
|
130
|
+
error, a `--filter` that matches nothing).
|
|
131
|
+
- `saffron eject` warns when the project's own `playwright.config` records
|
|
132
|
+
a trace and the exported tests read secrets, `{env:...}` values or
|
|
133
|
+
credentials the API hands out, which Playwright's trace keeps unmasked
|
|
134
|
+
(SAF-200). It no longer only checked on the first eject, and it leaves
|
|
135
|
+
the file as it is. A config it writes leaves the trace out for
|
|
136
|
+
captured credentials too.
|
|
137
|
+
- A pending proposal stays with the Outline row whose values it was
|
|
138
|
+
recorded on when a row is inserted above it or the rows are reordered
|
|
139
|
+
(SAF-201), where another row used to take its "(example N)" and record
|
|
140
|
+
over it. `saffron status` and the plugins name that row as the rows
|
|
141
|
+
stand now, and the review page shows that row's pictures, where they
|
|
142
|
+
used to name and show the row that took its place.
|
|
143
|
+
- An API credential is masked wherever it could still show: a token a
|
|
144
|
+
polled response handed out before the one waited for, a token a later
|
|
145
|
+
capture replaced (a poll that rotates it, then quotes the old one as
|
|
146
|
+
`previous_token`), and an error raised while a request is built (a body
|
|
147
|
+
value that is no number quoting the captured token it resolves to). The
|
|
148
|
+
exported test learns a token from every polled response too, where it
|
|
149
|
+
learnt only the last one's and an error quoted the one before.
|
|
150
|
+
- A compiled `Then` keeps a quoted value as written: `the /name field is
|
|
151
|
+
"Ada Lovelace"` checks for both spaces, where it used to check for one.
|
|
152
|
+
- `saffron accept <file>#<revision> --step-set` refuses a proposal replaced
|
|
153
|
+
since it was listed, as a whole accept does, where it took the
|
|
154
|
+
replacement's steps unseen.
|
|
155
|
+
- A request sent several times at once waits for every copy when one gets
|
|
156
|
+
no response, and keeps the copies that answered in the transcript, where
|
|
157
|
+
the failure came first and a late write went unrecorded.
|
|
158
|
+
- An exported test extracts an integer beyond JavaScript's precision as the
|
|
159
|
+
digits the API sent, as replay does, where it rounded an int64 id into
|
|
160
|
+
another record's.
|
|
161
|
+
- Each API response is let go once read, in replay and in exported tests,
|
|
162
|
+
so a long poll no longer holds every response it read until the scenario
|
|
163
|
+
ends.
|
|
164
|
+
- An exported text check reads the copy replay reads, a shown one after a
|
|
165
|
+
hidden copy with the same text (a mobile menu's), where it read the
|
|
166
|
+
hidden one and failed.
|
|
167
|
+
- A step naming its column for the statuses of copies sent at once
|
|
168
|
+
(`refused with <refusal>`) records them as that column, as it records a
|
|
169
|
+
single status, so the other rows replay where they were refused as
|
|
170
|
+
recorded for one row.
|
|
171
|
+
- A JSON check reads a number JavaScript cannot hold as written as a
|
|
172
|
+
number, written as the API sent it: a `min` or `max` compares it exactly
|
|
173
|
+
with the bound as written, in any notation (`9007199254740993`,
|
|
174
|
+
`9007199254740993.0` and `9.007199254740993e15` are all over a maximum
|
|
175
|
+
of `9007199254740992`, `9007199254740991.1` over one of
|
|
176
|
+
`9007199254740991`, and `1e9999999999` over `1`, where it stopped the
|
|
177
|
+
run), and a `pattern` (which reads text) does not hold on it. An `equals` still checks its digits, as the error for a typed
|
|
178
|
+
number advises.
|
|
179
|
+
- A compiled `Then` leaves a number JSON cannot hold (`the /n field is
|
|
180
|
+
007`) to the agent, where it compiled a check that could never pass.
|
|
181
|
+
|
|
182
|
+
Fixed, from real-world use of 1.3.1:
|
|
183
|
+
|
|
184
|
+
- The report's pass rate never reads 100% while a scenario failed: 473 of
|
|
185
|
+
475 reads 99.58%, where it was rounded up to 100%. The dashboard's
|
|
186
|
+
suite health, recorded and tag shares read the same way, and 0% means
|
|
187
|
+
none. `passRate` in
|
|
188
|
+
`saffron status --json` and the history stays a whole number, now never
|
|
189
|
+
100 unless every scenario passed (99 for 473 of 475).
|
|
190
|
+
|
|
191
|
+
- An Outline row that fails in a step before any step uses its own values
|
|
192
|
+
(a lunch sold out in a shared checkout step) is blamed on the app's
|
|
193
|
+
state, blocked by the environment, where the report used to call it
|
|
194
|
+
the recording's fault and point at a column the step never read. A
|
|
195
|
+
value used in a step's data table or doc string counts as used.
|
|
196
|
+
- A pending proposal stays with its row when a column is added to the
|
|
197
|
+
Examples: only the columns its row had tell rows apart, where every row
|
|
198
|
+
was called another row's and none recorded again.
|
|
199
|
+
- The sensitivity check judges a check against the action before a step
|
|
200
|
+
that only reads (a captured value, a watch armed), as it already did
|
|
201
|
+
for a wait: reading a booking number is not what confirms a payment.
|
|
202
|
+
- `api_contract` (the MCP tool) and the agent's contract summary list a
|
|
203
|
+
request body's alternatives with their fields (`one of: { ... } | { ...
|
|
204
|
+
}`) and an `allOf`'s parts as one body, where they said `any`.
|
|
205
|
+
- The Saffron skill's syntax reference describes the negative and exact
|
|
206
|
+
URL checks 1.3.1 added, where it said there were none.
|
|
207
|
+
- A recorded API step no longer keeps the agent's failed attempts
|
|
208
|
+
(SAF-202). A request whose body the pinned contract refuses is not
|
|
209
|
+
recorded, even when the API answered 2xx, unless the step expects a
|
|
210
|
+
refusal. The body is judged as sent, its tokens resolved: `{date+1}` as
|
|
211
|
+
a date, and `{captured:id}` as the text it sends, so an integer field
|
|
212
|
+
takes the number itself. A
|
|
213
|
+
request sent again to the same operation and expected answer right
|
|
214
|
+
after the step's last one takes its place, unless the step asks for
|
|
215
|
+
several ("twice", "two customers", "for each", "once more"), the earlier
|
|
216
|
+
one sent a value the step names that this one does not ("the currency
|
|
217
|
+
to SEK and guest names on" is two writes), this one sends the same
|
|
218
|
+
fields with other values the step does not name ("a mug and a hat"), or
|
|
219
|
+
both carry the same idempotency key. A value counts as named in any
|
|
220
|
+
case and in the step's data table or doc string, and as sent in a form
|
|
221
|
+
or the query too (`?status=draft`, then `?status=archived`); a step
|
|
222
|
+
with a table of several rows keeps every request.
|
|
223
|
+
- A `Then` that names a section in plain words (`the confirmation Summary
|
|
224
|
+
shows the package`, `the order card shows Paid`) has its checks scoped
|
|
225
|
+
to that section, as a quoted name already did (SAF-203): a check that
|
|
226
|
+
reads the whole page is refused while recording, instead of passing on
|
|
227
|
+
text that was there before the action.
|
|
228
|
+
- An Outline whose recorded row held one value in two columns
|
|
229
|
+
(`currency` and `display` both `NOK`) no longer refuses the other rows
|
|
230
|
+
for a step that names only one of them (`I choose "<currency>"`): the
|
|
231
|
+
recorder writes the column the step names. A step naming both, or
|
|
232
|
+
neither, still records again on a row that types them apart.
|
|
233
|
+
- A request that sends its header's credential in the body or URL as well
|
|
234
|
+
(`Authorization: Bearer {env:KEY}` and a body field `apiKey`) is refused
|
|
235
|
+
while recording, unless the step names both the body and the header.
|
|
236
|
+
Only an auth header counts: `X-Auth-Token`, not `X-Author` or
|
|
237
|
+
`X-Auth-User`.
|
|
238
|
+
- A "not shown" check naming a control by the start of its name (a
|
|
239
|
+
textbox `Firstname`, where the page shows `Firstname Required`) gets an
|
|
240
|
+
insensitive check note saying no element has that name, and naming the
|
|
241
|
+
control shown before the step.
|
|
242
|
+
- A check that finds its element by the text it checks (an amount found as
|
|
243
|
+
the text `SEK 495.00` anywhere on the page) is refused while recording:
|
|
244
|
+
it holds wherever that text shows. The agent finds the field the step
|
|
245
|
+
names, or records a visible check when the step only says the text is
|
|
246
|
+
shown.
|
|
247
|
+
|
|
248
|
+
Docs
|
|
249
|
+
|
|
250
|
+
- **How a recording is checked.** A new section of the guide (§24), and a
|
|
251
|
+
page of its own on the site, maps every check Saffron runs on a
|
|
252
|
+
recording before it is filed, in order: what each note means, which ones
|
|
253
|
+
hold a proposal from `accept --all` and when to accept it by file, what
|
|
254
|
+
is on by default, and how to turn on stability proofs.
|
|
255
|
+
|
|
256
|
+
Editors (VS Code extension 0.7.8, JetBrains plugin 0.7.9)
|
|
257
|
+
|
|
258
|
+
- Both editors mark `Examples: {data: roles}` or `{Data:roles}` with no
|
|
259
|
+
table under it as an error where it is written, as the runner stops on
|
|
260
|
+
it: VS Code in every project, JetBrains through the runner's language
|
|
261
|
+
server. Neither marks one inside a doc string or with a description
|
|
262
|
+
before its table.
|
|
263
|
+
- VS Code: the "Proposals pending review" row asks before it runs
|
|
264
|
+
`saffron accept --all`.
|
|
265
|
+
- JetBrains: Accept All and an accept or reject of selected proposals no
|
|
266
|
+
longer become the selected run configuration, so the Run button never
|
|
267
|
+
repeats them.
|
|
268
|
+
|
|
269
|
+
## 1.3.1: Checks for what vanishes, URL and style checks, and a StepSet accepted on its own
|
|
270
|
+
|
|
271
|
+
Released 2026-11-02.
|
|
272
|
+
|
|
273
|
+
Runner
|
|
274
|
+
|
|
275
|
+
Upgrading from 1.3.0:
|
|
276
|
+
|
|
277
|
+
- A recording with a negative or exact URL check, a check of a message
|
|
278
|
+
that vanishes, a computed-style check, or a value captured earlier
|
|
279
|
+
inside the element a step looks for is cache schema version 14, 15, 16
|
|
280
|
+
or 17, which 1.3.0 and earlier refuse ("Update saffron-ai"). An Outline
|
|
281
|
+
recording whose cell is a `{data:...}` value holding a date is version
|
|
282
|
+
18. Upgrade every machine and CI job that replays the suite together.
|
|
283
|
+
- `saffron.config.json` refuses a quoted number (`"actionTimeoutMs":
|
|
284
|
+
"10000"`), a negative one, or a `screenshot` other than `off`,
|
|
285
|
+
`viewport` or `full-page`, where it used to take them as they were.
|
|
286
|
+
- A row, data or captured value inside an API request's path is
|
|
287
|
+
percent-encoded where it would change the URL (`#`, `?`, a space; `&`,
|
|
288
|
+
`=` and `+` in the query): `A#1` is sent as `A%231`, no longer cut at
|
|
289
|
+
the `#`. A `/` and an existing `%20` are sent as they are.
|
|
290
|
+
- A `*` that opens a scenario after a Background ending in a `Then` is an
|
|
291
|
+
action now, as Cucumber reads it, not a check: a recording of such a
|
|
292
|
+
step records once more.
|
|
293
|
+
- A data table or doc string written under a `StepSet <name>` call is an
|
|
294
|
+
error naming the file and line, where it used to be dropped.
|
|
295
|
+
- With `reset` commands, recordings run one at a time when `agentWorkers`
|
|
296
|
+
is above 1 and the agent is on, and the run says so: one session's
|
|
297
|
+
reset would wipe the data another is recording on. Replays still use
|
|
298
|
+
`workers` as before.
|
|
299
|
+
- `strict`, `verifyProposals` and `reuseSteps` in `saffron.config.json`
|
|
300
|
+
take `true` or `false` without quotes; a quoted `"false"` used to read
|
|
301
|
+
as on.
|
|
302
|
+
- In an Outline, a number or boolean a request body or JSON check holds
|
|
303
|
+
only by coincidence with a row's value (a field not named like the
|
|
304
|
+
column, or a step not naming it) is kept as recorded and binds the
|
|
305
|
+
recording to its row, where it used to be sent and checked as each
|
|
306
|
+
row's value. Other rows are refused with a message until the field is
|
|
307
|
+
named like the column or the step names it. A constant beside a column
|
|
308
|
+
the step writes (`active: true` beside `admin: <admin>`) is kept as
|
|
309
|
+
recorded without binding the row.
|
|
310
|
+
- The recorder refuses checks that cannot fail: a URL check every URL
|
|
311
|
+
holds (`/`), a text, style, download or response pattern that matches
|
|
312
|
+
any text (`.*`), and a watch for a message other than the one the
|
|
313
|
+
`Then` names.
|
|
314
|
+
- A watched message counts as seen only when it appears after the watch
|
|
315
|
+
was armed (an element already showing it must stop showing it once),
|
|
316
|
+
and a seen check uses its watch up: a second check of the same message
|
|
317
|
+
needs a watch of its own. A recording that relied on a message left on
|
|
318
|
+
screen goes red and heals.
|
|
319
|
+
- `saffron diff --json` prints a list for one proposal too: a script that
|
|
320
|
+
read an object for one proposal reads the first entry now.
|
|
321
|
+
- SIGTERM stops a run as Ctrl-C does and exits 143, not 2.
|
|
322
|
+
- `--workers 0`, `--agent-workers 0` and a worker count that is not a
|
|
323
|
+
whole number exit 2 instead of running one worker.
|
|
324
|
+
- `init --examples` no longer writes into an existing
|
|
325
|
+
`saffron.config.json`.
|
|
326
|
+
|
|
327
|
+
- **From real-world use of Saffron.**
|
|
328
|
+
- A data value an API request takes through its step's table (`|
|
|
329
|
+
lastName | {data:signup.lastName} |`, sent as `<table:lastName>`) can
|
|
330
|
+
change and the recording replays at zero AI: the freshness check
|
|
331
|
+
follows the table cell to the data it names, and a header name HTTP
|
|
332
|
+
defines (`User-Agent`) is never taken for an old value ("User").
|
|
333
|
+
(SAF-177)
|
|
334
|
+
- A URL check with no URL is refused while recording, saying that a URL
|
|
335
|
+
check reads what the URL contains, and that there is no "does not
|
|
336
|
+
contain" or "is exactly" URL check yet: four such checks had been
|
|
337
|
+
recorded empty, and only the paid proof refused them. (SAF-178)
|
|
338
|
+
- `saffron run --plan --json` prints only the JSON on stdout, with a
|
|
339
|
+
shard too: the `shard 1/2 ...` line goes to stderr. (SAF-179)
|
|
340
|
+
- Fewer false sensitivity notes: a check is judged against the last step
|
|
341
|
+
that acted on the page, a step that only waits passed over; a
|
|
342
|
+
StepSet's exit guard is not probed against the set's own last action,
|
|
343
|
+
as it says the whole set got where it should (a set of checks only is
|
|
344
|
+
probed against the step before it); and a step that reads a whole
|
|
345
|
+
body, page or message gets no `broad check:` note. (SAF-180)
|
|
346
|
+
- From a review of these fixes: `--plan --json` also keeps the note on
|
|
347
|
+
Examples rows that share a recording off stdout, and an empty shard's
|
|
348
|
+
plan is an empty JSON plan, with no shard report written; a StepSet
|
|
349
|
+
used twice in a row has each use's exit guard left unprobed; an old
|
|
350
|
+
data value written into a body key or a custom header's name, whole or
|
|
351
|
+
in part, still makes the recording stale; "the body of a card", "the
|
|
352
|
+
entire row" and a card's body are not read as the whole page; and a URL
|
|
353
|
+
check's text is recorded trimmed. A header's value is still searched
|
|
354
|
+
for an old data value (`Authorization: Bearer <old token>`); only the
|
|
355
|
+
header names HTTP defines are left out, and a header an extract reads
|
|
356
|
+
by name is not taken for data.
|
|
357
|
+
|
|
358
|
+
- **From a three-round review of the runner.**
|
|
359
|
+
- No false green from a blank row value: a role name pattern, a response
|
|
360
|
+
URL, a JSON field or header pattern, a watched message and a style
|
|
361
|
+
pattern that a blank value leaves matching more than they mean are
|
|
362
|
+
refused, as text and URL checks already were. An expected status left
|
|
363
|
+
blank is refused, never read as any 2xx.
|
|
364
|
+
- Outline rows never record over each other's proposal: a row whose
|
|
365
|
+
pending proposal was recorded for another row is red, with a committed
|
|
366
|
+
recording too, and a proposal from another machine never turns a
|
|
367
|
+
failing protected `Then` yellow.
|
|
368
|
+
- A fixture takes values only from a proof that passed, and a proposal
|
|
369
|
+
from another machine is never replayed for its values.
|
|
370
|
+
- A scenario whose run throws (a busy proposal lock, a full disk) is that
|
|
371
|
+
scenario's red; the run goes on and writes its report. Reports, shard
|
|
372
|
+
reports and the history are written whole, never half.
|
|
373
|
+
- A contract check repaired by a heal keeps every operand of the check:
|
|
374
|
+
a URL, pattern or attribute changed is review, not repair.
|
|
375
|
+
- Secrets: a value set with a trailing newline is masked as the page
|
|
376
|
+
shows it, trimmed; credential headers and cookies in a trace are
|
|
377
|
+
replaced whole (`Authorization: Basic ...`); names written as one word
|
|
378
|
+
(`authtoken`, `JSESSIONID`) count as credentials; a header the
|
|
379
|
+
transcript keeps is judged by the same rule as the trace.
|
|
380
|
+
- A pattern holding a harmless escape (`\#`, `\:`) next to a row value
|
|
381
|
+
no longer fails as an invalid regular expression.
|
|
382
|
+
- `{captured:...}` in the element a step looks for is the captured value,
|
|
383
|
+
and a retried "message was shown" check can still see the message.
|
|
384
|
+
- API scenarios: YAML merge keys (`<<: *base`) in a contract are applied,
|
|
385
|
+
the response schema is chosen by the response's content type,
|
|
386
|
+
`differs` compares lists and objects, `the /x field is null` compiles,
|
|
387
|
+
a phrase-compiled `Then` serves every Outline row, and with
|
|
388
|
+
`api.transcript: "off"` no error quotes a response body.
|
|
389
|
+
- Recording: a multi-line typed text keeps its `{captured:...}`, a
|
|
390
|
+
`{unique:...}` inside an Examples cell is typed resolved, a feature
|
|
391
|
+
edit writes `<plan>` only for the whole word (`Production` stays),
|
|
392
|
+
and Antigravity stops when the run aborts.
|
|
393
|
+
- `saffron eject` writes tests never looser than replay: an exact path
|
|
394
|
+
or style is compared whole, enabled and disabled checks need the
|
|
395
|
+
element shown, `differs` compares amounts, a not-visible check reads
|
|
396
|
+
every locator at once, timed waits are capped, the device's own
|
|
397
|
+
engine is dropped, and Playwright's trace is left out when the tests
|
|
398
|
+
read `{env:...}` values, since it would record them unmasked.
|
|
399
|
+
- CLI: `run --plan` validates its flags as `run` does, `coverage --write
|
|
400
|
+
--json` prints only JSON, the HTML report escapes every proposal
|
|
401
|
+
field, and `init` sets up the rest before naming an MCP file it
|
|
402
|
+
cannot merge.
|
|
403
|
+
- Amounts and dates: `(1.5 kg)` is positive, and a date range that ends
|
|
404
|
+
before it starts is refused.
|
|
405
|
+
|
|
406
|
+
- **From a second two-round review of the runner.**
|
|
407
|
+
- Outline rows: `accept --propagate` no longer copies a healed locator
|
|
408
|
+
holding its Outline row's value into other recordings, and `saffron
|
|
409
|
+
status` marks an Outline stale when any row would record again, naming
|
|
410
|
+
the row.
|
|
411
|
+
- Secrets: a value set with a trailing newline is masked where the page
|
|
412
|
+
holds it trimmed (in a URL, in another case, escaped in a selector or
|
|
413
|
+
glued into a word), and a warning about an unhandled rejection masks
|
|
414
|
+
the credentials a session captured.
|
|
415
|
+
- Recording: a target with nothing to find the element by is refused, a
|
|
416
|
+
`saffron_step` whose text names another step is refused so actions are
|
|
417
|
+
never filed a step late, the first `saffron_done` verdict stands, a
|
|
418
|
+
pass past the time budget that leaves a step unrecorded is a failure,
|
|
419
|
+
and a Claude session stopped at its turn limit says to raise
|
|
420
|
+
`maxTurns`; Claude token counts include compaction.
|
|
421
|
+
- Files and processes: `prune` keeps a merge-conflicted recording of a
|
|
422
|
+
live scenario outside `features` and the recording of an Outline whose
|
|
423
|
+
Examples have no rows for now; a scenario name of any length can be
|
|
424
|
+
saved, and recordings saved under the whole name are still found;
|
|
425
|
+
Ctrl-C stops a running `reset`; a `hookTimeoutMs` past a timer's limit
|
|
426
|
+
no longer ends every hook at once; a hook timed out on Windows ends with
|
|
427
|
+
its child processes; a run killed while writing history no longer
|
|
428
|
+
spoils the next line; a browser that fails to close or a history file
|
|
429
|
+
that cannot be written no longer costs the run its report; writes ride
|
|
430
|
+
out a file briefly locked on Windows; a failed browser close leaves no
|
|
431
|
+
video folder behind; `init` git-ignores the temp and lock files a
|
|
432
|
+
killed run leaves.
|
|
433
|
+
- `saffron eject` writes tests never looser than replay: a schema check
|
|
434
|
+
finds its operation with the row's value in the path, a response with
|
|
435
|
+
a schema per JSON media type is refused, a blank expected status and
|
|
436
|
+
the NAT64 and IPv4-compatible link-local forms are refused, attribute
|
|
437
|
+
and style checks read the element replay reads, a blank captured value
|
|
438
|
+
in a checked text fails, a dialog answer from a row is that row's, a
|
|
439
|
+
whitespace-only text pattern is fixme, and header patterns compile as
|
|
440
|
+
replay compiles them.
|
|
441
|
+
- Status, plan and coverage: a feature in another Gherkin language shows
|
|
442
|
+
its name, `run --plan` counts a feature's fixture, `coverage` refuses a
|
|
443
|
+
suite with a file it cannot parse and `verify` says so, and a scenario
|
|
444
|
+
named like an Outline row keeps its own last result.
|
|
445
|
+
- Editors and reports: the language server reads doc string lines as
|
|
446
|
+
data, not StepSet calls, and the CI run panel shows a cost under a cent
|
|
447
|
+
as the report does.
|
|
448
|
+
|
|
449
|
+
- **From a two-round review of the runner.**
|
|
450
|
+
- No false green from another row or a leftover: an Outline cell that
|
|
451
|
+
reads a date through a data value has its day checked as a date cell
|
|
452
|
+
does; a step reused by swapping a quoted value no longer offers the old
|
|
453
|
+
value's other recordings; a message already on screen before an action
|
|
454
|
+
no longer counts as seen; a capture pattern a blank row value leaves
|
|
455
|
+
open is refused.
|
|
456
|
+
- Recording types a captured value as plain text, as replay does, and
|
|
457
|
+
refuses a browser call whose captured value holds an `{env:...}`
|
|
458
|
+
token rather than type the secret.
|
|
459
|
+
- Checks without a frame also find what an iframe attached after the
|
|
460
|
+
check began shows; checks with fallback locators end at their timeout;
|
|
461
|
+
a drag finds its drop zone by its fallbacks; the heal guard runs a
|
|
462
|
+
check on the tab an earlier action moved to or opened.
|
|
463
|
+
- Seeded recording keeps a watch armed in one stretch for its `Then` in
|
|
464
|
+
the next, and seeded steps and the sensitivity check read values
|
|
465
|
+
another step's table or doc string gives.
|
|
466
|
+
- API: a schema check finds the operation for a path with a parameter
|
|
467
|
+
inside a segment (`/files/{name}.json`); a row value after a captured
|
|
468
|
+
URL that brings its own query is encoded as a query value, in replay
|
|
469
|
+
and in exported tests; with no agent, an Outline's rows file one
|
|
470
|
+
contract re-check, not one each; `api pin` passes over a fallback URL
|
|
471
|
+
that does not answer with OpenAPI 3.
|
|
472
|
+
- Proposals: `accept` notices a hand edit of a proposal's API contract
|
|
473
|
+
fingerprints, `accept --step-set` refuses to replace another StepSet
|
|
474
|
+
whose name makes the same file name, and `run` reports two StepSets or
|
|
475
|
+
a fixture and a scenario whose recordings would share a file; an
|
|
476
|
+
accepted heal no longer copies a captured value into other scenarios;
|
|
477
|
+
a "capture differs" note leaves credentials out.
|
|
478
|
+
- Reports: a password holding an `@` in a URL on the command line is
|
|
479
|
+
hidden whole, `report --merge` refuses a shard it cannot show, naming
|
|
480
|
+
the field, an exported or imported bundle's report names only the
|
|
481
|
+
evidence the bundle carries, and the trace page marks a request the
|
|
482
|
+
agent did not record.
|
|
483
|
+
- CLI and editors: `saffron author` fails with the provider's error
|
|
484
|
+
instead of writing it as the draft; `init` refuses an instructions
|
|
485
|
+
block with no end marker instead of deleting the text after it; a
|
|
486
|
+
config file holding something other than an object is refused by
|
|
487
|
+
name; an exported timed wait written with a row value waits as long
|
|
488
|
+
as replay; feature edits apply to steps whose keyword is several
|
|
489
|
+
words (`Étant donné que`); the language server started by an editor
|
|
490
|
+
reads a project with no `features` key as the default folder.
|
|
491
|
+
|
|
492
|
+
- **A `Then` is read-only while recording.** A check refuses a browser
|
|
493
|
+
action that enters, presses keys, picks, ticks, uploads, drags or
|
|
494
|
+
navigates, and a click on a button that submits or commits (Log in,
|
|
495
|
+
Save, Submit, Confirm and the like), before the browser runs it: the
|
|
496
|
+
agent is told the step is a check and to fail it, suggesting the action
|
|
497
|
+
as a `When`, if it does not hold as the page stands. Before, such a
|
|
498
|
+
click was recorded and proved, and held for review afterwards; a login
|
|
499
|
+
suite had a check click Log in, and others submit again to bring back a
|
|
500
|
+
toast that had vanished. A click that opens what is checked (a menu, a
|
|
501
|
+
tab) is still recorded and held as before, and proposals filed before
|
|
502
|
+
this stay held from `accept --all`. (SAF-181)
|
|
503
|
+
|
|
504
|
+
- **A "not shown" check must see something disappear.** The sensitivity
|
|
505
|
+
check after a proof runs each "not shown" check in a `Then` that holds
|
|
506
|
+
other checks on its own, before the step it follows; one that holds
|
|
507
|
+
there too (its element was never on the page, as with an invented
|
|
508
|
+
selector) gets an `insensitive check:` note naming it, held from
|
|
509
|
+
`accept --all`. Before, the step's other checks failing there hid it,
|
|
510
|
+
and it stayed green whatever the page showed. The recording agent is
|
|
511
|
+
also told to target the element a named field labels (the booker, not
|
|
512
|
+
any paragraph showing the name) and only elements it saw for a "not
|
|
513
|
+
shown" check. (SAF-183)
|
|
514
|
+
|
|
515
|
+
- **A StepSet's own data value stays its own in an Outline.** A step that
|
|
516
|
+
enters a `{data:...}` value itself, while not naming the column (a
|
|
517
|
+
StepSet typing `{data:guests.first.surname}`), no longer has the rows
|
|
518
|
+
refused because one row's cell reads the same: the email row of a login
|
|
519
|
+
suite was refused for it. A value a step only checks is still refused
|
|
520
|
+
for the rows that differ, as the page may show it from the row.
|
|
521
|
+
(SAF-182)
|
|
522
|
+
|
|
523
|
+
- **Trial navigations stay out of the recording.** A navigation the
|
|
524
|
+
agent leaves for the same page with another query or fragment, before
|
|
525
|
+
doing anything there, is discarded and noted in the proposal: a login
|
|
526
|
+
suite kept each query format the agent tried as a replay action. A
|
|
527
|
+
navigation to another page, or one acted on, is kept. (SAF-191)
|
|
528
|
+
|
|
529
|
+
- **A part of a name the agent wrote replays.** A click through a locator
|
|
530
|
+
the agent wrote, naming a part of the element's name, is recorded with
|
|
531
|
+
the full name the page it saw last shows for the one element of that
|
|
532
|
+
role holding it: replay matches a name whole, and timed out on "Sign
|
|
533
|
+
in". Where the page showed none or several, the proposal notes it
|
|
534
|
+
while recording, before the proof. (SAF-184)
|
|
535
|
+
|
|
536
|
+
- **Negative and exact URL checks.** A URL check can now say that the
|
|
537
|
+
URL does not contain a value, or that its path is exactly one, recorded
|
|
538
|
+
from the step's own words: before, a URL check was containment only,
|
|
539
|
+
and steps saying the URL no longer holds the login could not be
|
|
540
|
+
recorded. Replay and `saffron eject` check them alike. A recording
|
|
541
|
+
holding one is cache schema version 14, so an older runner refuses it
|
|
542
|
+
instead of checking containment. (SAF-186)
|
|
543
|
+
|
|
544
|
+
- **Checks for messages that vanish.** A toast or error flash gone
|
|
545
|
+
before its `Then` runs can be checked: the agent arms a watch
|
|
546
|
+
(`saffron_watch`) in the `When` step before the action that shows it,
|
|
547
|
+
and the `Then` checks that it was seen since. Replay watches the same
|
|
548
|
+
way. Before, a login error toast vanished before the check, and
|
|
549
|
+
recordings submitted again inside the `Then` to bring it back. A
|
|
550
|
+
recording holding a watch is cache schema version 15; `saffron eject`
|
|
551
|
+
writes its scenario as `test.fixme` for now. (SAF-185)
|
|
552
|
+
|
|
553
|
+
- **Computed-style checks.** A step can check an element's computed CSS
|
|
554
|
+
property (an invalid input's `border-color` is `rgb(220, 38, 38)`), or
|
|
555
|
+
match it to a pattern: before, only classes, attributes and the visible
|
|
556
|
+
error could be checked. Replay and `saffron eject` check it alike, in
|
|
557
|
+
Chromium, Firefox and WebKit. A recording holding one is cache schema
|
|
558
|
+
version 16. `saffron eject` also writes the negative and exact URL
|
|
559
|
+
checks now, which it had written as `test.fixme`. (SAF-187)
|
|
560
|
+
|
|
561
|
+
- **A navigation to a captured URL stays the capture.** When the agent
|
|
562
|
+
goes to `{captured:signin}` and the value is an address on the base URL,
|
|
563
|
+
the recording keeps `{captured:signin}`: before, it kept the day's path
|
|
564
|
+
relative to the base, and replay went to the old link instead of the one
|
|
565
|
+
its own request got. (SAF-188)
|
|
566
|
+
|
|
567
|
+
- **Accept one StepSet from a proposal.** `saffron accept <file>
|
|
568
|
+
--step-set "Sign in"` files only that set's steps, which its proof
|
|
569
|
+
replayed green, for every other scenario to reuse at zero AI, while the
|
|
570
|
+
proposal and its failing check stay pending. Before, a failed check kept
|
|
571
|
+
a verified sign-in from being reused, and each scenario paid to record it
|
|
572
|
+
again. A proposal whose proof failed records how many steps passed before
|
|
573
|
+
it (`meta.provedSteps`), and a set the proof did not get through is
|
|
574
|
+
refused. (SAF-189)
|
|
575
|
+
|
|
576
|
+
- **Reuse picks the recording the page's state fits.** A step recorded in
|
|
577
|
+
two page states (a promo field opened first, or found open) keeps both
|
|
578
|
+
recordings, and seeding replays the one whose controls the page shows
|
|
579
|
+
now: before, it always took the newest, which closed a field already open
|
|
580
|
+
and paid the agent to record the step again. (SAF-190)
|
|
581
|
+
|
|
582
|
+
- **CLI and files, from the 1.0 review.** (SAF-133)
|
|
583
|
+
- A scenario with no steps is red, saying so, and starts no agent
|
|
584
|
+
session: it had recorded nothing and passed.
|
|
585
|
+
- A broken link or a folder named `*.json` under `.saffron/proposals`
|
|
586
|
+
no longer stops `status`, `diff` and the end of a run.
|
|
587
|
+
- `saffron diff --json` prints a list for one proposal too, as it does
|
|
588
|
+
for several.
|
|
589
|
+
- `accept`, `reject` and `diff` find a project-relative file (as
|
|
590
|
+
`status` prints it) under `--project-root`, not only from the current
|
|
591
|
+
folder.
|
|
592
|
+
- `accept --with-feature-edit` edits a step line that ends in spaces.
|
|
593
|
+
- `saffron status` prints an unreadable recording's warning on a line of
|
|
594
|
+
its own.
|
|
595
|
+
- `saffron init` keeps a skill you edited under `.saffron/backups/skills/`
|
|
596
|
+
before installing the shipped one.
|
|
597
|
+
- Already fixed and checked: an empty run exits 2 naming why, a StepSet
|
|
598
|
+
file that does not parse is named, recordings are written atomically,
|
|
599
|
+
a CRLF file takes feature edits, and status counts an unreadable
|
|
600
|
+
recording apart.
|
|
601
|
+
|
|
602
|
+
- **A check's selector is tried on the page while recording.** A check
|
|
603
|
+
whose CSS selector matches nothing on the page the agent sees is refused
|
|
604
|
+
then, at no cost, naming the likely cause (a link styled as a button is
|
|
605
|
+
an `<a>`): before, only the paid proof caught `p:has(button)`. The other
|
|
606
|
+
half of this story, a `broad check:` note for a check held by a whole
|
|
607
|
+
card, shipped in 1.3.0. (SAF-171)
|
|
608
|
+
|
|
609
|
+
- **Exact `Then` phrases need no agent.** In an API scenario, `Then the
|
|
610
|
+
response should answer 404`, `Then the /status field is active` and
|
|
611
|
+
`Then the response matches the Customer schema` are recorded as those
|
|
612
|
+
checks without the agent, which records only the steps around them. Any
|
|
613
|
+
other wording goes to the agent as before, never guessed. An option, not
|
|
614
|
+
a required syntax. (SAF-162)
|
|
615
|
+
|
|
616
|
+
- **Live progress, and a time budget per agent session.** While an agent
|
|
617
|
+
records, the run prints on stderr each step it records and each proof as
|
|
618
|
+
it ends (`· Checkout: recording step 41 of 92: ...`, `· Checkout: proof
|
|
619
|
+
1 passed`): a 92-step recording ran 340 seconds with no word until it
|
|
620
|
+
finished. `maxSessionMinutes` (`--max-session-minutes`) ends one session
|
|
621
|
+
that has gone on that long: its tools refuse, and it ends unfinished
|
|
622
|
+
with the agent's account of where it got to. (SAF-194)
|
|
623
|
+
|
|
624
|
+
- **Row proofs show their time and each row's latest result.** `saffron
|
|
625
|
+
diff` and the review page say when an Outline's rows were proved, and
|
|
626
|
+
beside each row its latest result in the last run (`latest run: red at
|
|
627
|
+
...`, or `no run since`): a proposal's passing row proofs were read as
|
|
628
|
+
readiness while later runs of those rows timed out. (SAF-192)
|
|
629
|
+
|
|
630
|
+
- **A failed response is named beside the step it broke.** A step that
|
|
631
|
+
fails after one of the app's own requests reported failure (an error
|
|
632
|
+
status, or a JSON body with `success: false`, `ok: false` or an
|
|
633
|
+
`error`) says so in its error, which the console, the report and the
|
|
634
|
+
heal prompt all show: an HTTP 200 holding "overbooked" before a timeout
|
|
635
|
+
reaching checkout no longer reads like a locator problem. (SAF-193)
|
|
636
|
+
|
|
637
|
+
- **Outline rows, from the 1.0 review.** (SAF-128)
|
|
638
|
+
- A list check whose values come from `{data:...}` or `{env:...}` cells
|
|
639
|
+
("Alice, Bobby" from `{data:guests.a}` and `{data:guests.b}`) is tied
|
|
640
|
+
to its row, as one of plain cells is, so another row fails it with the
|
|
641
|
+
reason and no agent session: it was written as the row's `<column>`,
|
|
642
|
+
and every other row failed it and paid the agent again on every run.
|
|
643
|
+
- The guide says what a row-bound recording keeps of a short secret
|
|
644
|
+
read through an Examples cell, and why a PIN belongs elsewhere.
|
|
645
|
+
- Checked and kept with tests: a value inside a word of what a
|
|
646
|
+
Background step types (`test` in `qa-tester@example.com`) binds no row;
|
|
647
|
+
a date cell typed as ISO, a value seen only as a slug or in another
|
|
648
|
+
case, and a column naming a selector's role refuse the other rows
|
|
649
|
+
instead of recording again; a URL-encoded value is the row's
|
|
650
|
+
`<column>`; a `{data:...}` value naming an unset variable stops the run
|
|
651
|
+
before it starts; a short secret glued to other text is masked.
|
|
652
|
+
|
|
653
|
+
- **Visible text, from the 1.0 review.** (SAF-129)
|
|
654
|
+
- A check whose value is written on several lines (a two-line doc
|
|
655
|
+
string) holds where the page draws those lines, split by a `<br>` or
|
|
656
|
+
by blocks: it never matched "Line one<br>Line two", which read "Line
|
|
657
|
+
oneLine two". A value on one line still reads the text as one run, so
|
|
658
|
+
two blocks drawn apart never make it.
|
|
659
|
+
- A list check (`expectEach`) no longer passes on a hidden option, or
|
|
660
|
+
hidden text, that differs from a shown one only in case: the page's
|
|
661
|
+
lines in another case count only where CSS `text-transform` changes
|
|
662
|
+
them.
|
|
663
|
+
- A check on a large container is faster: 22 ms instead of 56 ms on a
|
|
664
|
+
4,000-row section. 0.9.9 takes 4 ms there, as it did not read what is
|
|
665
|
+
drawn; the rest is the cost of reading only drawn text.
|
|
666
|
+
- Kept as it is: an element without text whose first match is hidden
|
|
667
|
+
gives way to a displayed one, as a hidden mobile copy does to the
|
|
668
|
+
desktop one; a check that means one row's badge targets that row.
|
|
669
|
+
|
|
670
|
+
- **CLI, reports and the trace viewer, from the 1.0 review.** (SAF-130)
|
|
671
|
+
- `saffron diff` names the text a role with no name is found by:
|
|
672
|
+
`expect listitem with text "Milk" to be visible`, not every listitem.
|
|
673
|
+
- The trace viewer titles another browser session's step by the keyword
|
|
674
|
+
it was written with (`Given`, `And`), not by its kind, and a session
|
|
675
|
+
opens on a step of its own rather than on another session's first one.
|
|
676
|
+
- The results page embeds at most 20 MB of screenshots, failures first;
|
|
677
|
+
past that it names each file instead (300 failures made a 78 MB page).
|
|
678
|
+
- A feature reached through a linked folder's alias, or named in another
|
|
679
|
+
case on a case-insensitive disk, is the suite's feature, with its
|
|
680
|
+
recordings, not a new one.
|
|
681
|
+
- The MCP tools keep answering while `saffron.config.json` cannot be read,
|
|
682
|
+
on the config they last read, and say what is wrong with it.
|
|
683
|
+
- `init --examples` leaves an existing `saffron.config.json` as it is:
|
|
684
|
+
the examples' `retries`, `strict` and `trace` no longer change the
|
|
685
|
+
project's own runs, and a missing `baseURL` is named in a note.
|
|
686
|
+
- A bundle made in a git checkout with no remote is refused for that, not
|
|
687
|
+
as one made outside a checkout.
|
|
688
|
+
- SIGTERM stops a run as Ctrl-C does (teardown, the report of what
|
|
689
|
+
finished) and exits 143, instead of a red line and exit 2.
|
|
690
|
+
- A red scenario that replayed no step is labelled `not run` in the run
|
|
691
|
+
output, not `cache`.
|
|
692
|
+
- `--workers 0`, or a typo in `--agent-workers` or in the config's
|
|
693
|
+
`workers` or `agentWorkers`, exits 2 instead of running one worker.
|
|
694
|
+
- Fewer temporary folders are left behind by the test suite.
|
|
695
|
+
- Kept as they are: `saffron status` already marks an Examples row a
|
|
696
|
+
row-bound recording refuses (✗, with the reason), and the chronic hint
|
|
697
|
+
counts only reds that cost an agent session, which a refused row never
|
|
698
|
+
does. `saffron report --no-open` already writes the report without
|
|
699
|
+
opening a browser (1.3.0).
|
|
700
|
+
|
|
701
|
+
- **Language server and editor parity, from the 1.0 review.** (SAF-131)
|
|
702
|
+
- The language server reads data Examples under every Examples keyword
|
|
703
|
+
of the file's language (`Scenarios:`, `Exemples:`), as the parser
|
|
704
|
+
does, instead of flagging them; a bare `{data:file}` in a step is no
|
|
705
|
+
reference to the runner, so it gets no hover and no error; and the
|
|
706
|
+
`# language:` header is read with Gherkin's own pattern.
|
|
707
|
+
- Its completion rates a step after `As "<who>"` (a session an Examples
|
|
708
|
+
column names) unrecorded, as step reuse never seeds it.
|
|
709
|
+
|
|
710
|
+
- A text check that fails on a value the page holds but does not show says
|
|
711
|
+
where it sits: `Timed out waiting for text "Total: 42 NOK": the page holds
|
|
712
|
+
it, but not shown (inside a closed <details>)`, or in an element with
|
|
713
|
+
`display: none`, the `hidden` attribute, `visibility: hidden` or
|
|
714
|
+
`content-visibility: hidden`, naming the element. A total that is gone and one behind a panel that did not open read apart. Text
|
|
715
|
+
the page does not hold reads as before. (SAF-195)
|
|
716
|
+
- A text check on a large container is fast again: 3.5 ms on a
|
|
717
|
+
4,000-row container (0.9.9 took 2.4 ms, reading hidden text too). The value is looked for in the container's text first,
|
|
718
|
+
and only the text it covers is asked whether it is drawn; anything that
|
|
719
|
+
question cannot settle (a textarea, a dropdown, a shadow host) is read
|
|
720
|
+
the full way, so the check passes on exactly what it passed on before.
|
|
721
|
+
(SAF-129)
|
|
722
|
+
- A range of days the page shows with its year (`20–22 Oct 2026`,
|
|
723
|
+
`Oct 30 - Nov 1, 2026`, `31 Dec 2026–2 Jan 2027`, German
|
|
724
|
+
`20.-22.10.2026`), up to 62 days long, is recorded as a range token
|
|
725
|
+
(`{date+7–+9:D MMM YYYY}`), as a single day already was. A step that
|
|
726
|
+
names a stay by intent ("check-in in 7 days for 2 nights") now records
|
|
727
|
+
a check that follows the clock instead of a date frozen on the day of
|
|
728
|
+
recording. The agent is no longer told to match dates with a pattern,
|
|
729
|
+
which passes on any day. (SAF-124)
|
|
730
|
+
- The agent skill and the block `saffron init` writes into `AGENTS.md` and
|
|
731
|
+
`CLAUDE.md` say when a small supporting script is the right tool (only
|
|
732
|
+
for a value Saffron cannot express, asked for first) and how to keep it
|
|
733
|
+
trustworthy: expected values never taken from the output being checked,
|
|
734
|
+
run as a `setup` command writing git-ignored `data/` files, assumptions
|
|
735
|
+
stated, and a report of what could replace it. Run `saffron init` again
|
|
736
|
+
to update the block. (SAF-125)
|
|
737
|
+
- A video of each scenario's run: `"video": "retain-on-failure"` keeps
|
|
738
|
+
the video of every scenario that is not green, `"on"` every one, in
|
|
739
|
+
`video.webm` beside its screenshots; also `--video` for one run. Off
|
|
740
|
+
by default. It works in Chromium, Firefox and WebKit, and a heal or a
|
|
741
|
+
recording is in the video of the run it happened in; proof replays are
|
|
742
|
+
not filmed. A browser that cannot record (no ffmpeg) stops the run once,
|
|
743
|
+
at the start. A summary export carries no video. A video shows what is
|
|
744
|
+
typed, secrets included, so keep it off for scenarios that type them.
|
|
745
|
+
(SAF-51)
|
|
746
|
+
- The limits of the CI runs viewed into `.saffron/ci/` are settings:
|
|
747
|
+
`"ci": {"cacheRuns": 20, "cacheMB": 4096}`. The defaults stay 10 runs
|
|
748
|
+
and 1 GB. (SAF-121)
|
|
749
|
+
- Each feature edit a proposal suggests names the file and line it
|
|
750
|
+
rewrites, in `saffron diff` (`file` and `line` in `--json`), the review
|
|
751
|
+
page and the run report. A StepSet's step names the StepSet's
|
|
752
|
+
definition, where accept rewrites it for every scenario using the set,
|
|
753
|
+
and an edit to a Background step is marked skipped, as accept leaves it
|
|
754
|
+
to you. (SAF-31)
|
|
755
|
+
- `saffron status --json` lists each pending proposal's feature edits
|
|
756
|
+
(`featureEdits`, as `diff --json` gives them, with file and line), and
|
|
757
|
+
the VS Code and JetBrains panels show them in a proposal's details
|
|
758
|
+
without another command. (SAF-30)
|
|
759
|
+
- Each step a proposal changes is marked with its class: `repair` (only
|
|
760
|
+
how the step finds the same element again or waits for it: a fallback
|
|
761
|
+
selector, a timeout, a wait), `review` (what it does changed, an
|
|
762
|
+
element named another way, or one of Saffron's notes names it) or, for a
|
|
763
|
+
suggested feature edit, `protected` (the wording, which only you
|
|
764
|
+
apply). It shows in `saffron diff` (`adaptation` in `--json`, with
|
|
765
|
+
counts in `summary`), the review page and the run report, and the
|
|
766
|
+
guide has a section on the three classes. (SAF-32)
|
|
767
|
+
- From a review of this release: a step whose target is named by a
|
|
768
|
+
selector inside an iframe is `review` when the selector changes (the
|
|
769
|
+
frame names the iframe, not the element); the heal branch's pull request
|
|
770
|
+
body, the run report and the editors' trace panel class each step as
|
|
771
|
+
`saffron diff` does, reading the proposal's notes, and name the file and
|
|
772
|
+
line of each feature edit; a video with no viewport setting is filmed at
|
|
773
|
+
the page's own 1280x720, not Playwright's 800x800 fit; a range that
|
|
774
|
+
writes the month twice (`Oct 20 - Oct 22, 2026`), one ending exactly 31
|
|
775
|
+
days back, or one from before today to after it (`Room 7 - 22 Oct 2026`
|
|
776
|
+
is a room) is kept whole, never with only its end rewritten.
|
|
777
|
+
|
|
778
|
+
Editors (VS Code extension 0.7.7, JetBrains plugin 0.7.8)
|
|
779
|
+
|
|
780
|
+
- Both editors list the feature file edits a proposal suggests, each with
|
|
781
|
+
the file and line `saffron accept --with-feature-edit` rewrites and why
|
|
782
|
+
accept would skip one: VS Code in the proposal tooltip, JetBrains in the
|
|
783
|
+
Proposals tab details, read from `saffron status --json`. (SAF-30)
|
|
784
|
+
- VS Code: a `# language:` header Gherkin rejects is a plain comment; a
|
|
785
|
+
file a link inside the test folder leads back to is listed once; a
|
|
786
|
+
StepSet under a nested `saffron.config.json` in the test folder is
|
|
787
|
+
defined. (SAF-131)
|
|
788
|
+
- JetBrains: the Files tab fails on a folder it cannot read, as the runner
|
|
789
|
+
does, instead of hiding it; right-click Run reads the IDE's view of the
|
|
790
|
+
files, not the disk through links, each time the menu opens; a Windows
|
|
791
|
+
junction is walked as a link. (SAF-131)
|
|
792
|
+
- Both editors open a report path through a link outside the test folder
|
|
793
|
+
only when it really leads into the project, as the runner follows links
|
|
794
|
+
inside the test folder only. (SAF-131)
|
|
795
|
+
|
|
8
796
|
## 1.3.0: Recordings checked by machine, and a review every view agrees on
|
|
9
797
|
|
|
10
798
|
Released 2026-10-30.
|