saffron-ai 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +653 -0
- package/README.md +3 -5
- package/dist-pkg/cli.js +342 -283
- package/package.json +2 -2
- package/skills/saffron/SKILL.md +34 -7
- package/skills/saffron/references/config.md +2 -0
- package/skills/saffron/references/syntax.md +50 -8
- package/templates/playwright/saffron-helpers.ts +652 -0
- package/textmate/saffron/syntaxes/saffron.tmLanguage.json +12 -3
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,659 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.1.0: Secrets the AI agent never sees
|
|
9
|
+
|
|
10
|
+
Released 2026-10-22.
|
|
11
|
+
|
|
12
|
+
**Upgrading from 1.0.1.** Nothing to change in a project. What can be
|
|
13
|
+
different after `npm update`:
|
|
14
|
+
|
|
15
|
+
- Row value boundaries are stricter: a regex atom that may match nothing
|
|
16
|
+
or anything (`\W*`, `.+`, `[^,]+`), a lookaround, or a group edge or
|
|
17
|
+
alternation next to a row value (`Order <id>(?:st|)`) no longer drops
|
|
18
|
+
the value's boundary, and row values are matched in every selector-chain
|
|
19
|
+
container. A row value at the edge of a role's name, a label, a
|
|
20
|
+
placeholder, an attribute (`alt`, `title`, `[aria-label*=...]`, XPath
|
|
21
|
+
`contains(@href, ...)`), `:text(...)`, `:text-matches(...)`, an inner
|
|
22
|
+
`has` locator or a frame's selector is whole too, as in text: row `1`
|
|
23
|
+
never finds `Order 12`. Beside a digit, a decimal or thousands part
|
|
24
|
+
counts as part of the number (`1` is not in `1.5` or `1,000`) unless the
|
|
25
|
+
pattern writes it (`<n>\.\d{2}`), and a pattern with a quantifier right
|
|
26
|
+
after a row value (`<id>+`) is refused. Text a selector says must be
|
|
27
|
+
absent (`:not(...)`, XPath `not(...)`, `has-not-text`) is no longer
|
|
28
|
+
looked for as present.
|
|
29
|
+
A recording that passed in 1.0.1 because of that can turn red after
|
|
30
|
+
`npm update`; it never turns a failing check green. Re-record what fails
|
|
31
|
+
(`saffron run --rerecord <file>:<line>`).
|
|
32
|
+
- Recordings that use JSON response field checks, or single-quoted row
|
|
33
|
+
values in any part of a selector chain (an XPath part first too), are
|
|
34
|
+
format version 5. Saffron 1.0.1 refuses them by name, so a team updates
|
|
35
|
+
together.
|
|
36
|
+
- An Outline with `As "<who>"` steps recorded before 1.0.1, where a step
|
|
37
|
+
types a value of that person's own, records again once on another
|
|
38
|
+
person's row (red under `--no-agent`, saying why). The new recording
|
|
39
|
+
then refuses the other rows by name, as 1.0.1 recordings of that shape
|
|
40
|
+
do: give each person's values a column, or each person a Scenario.
|
|
41
|
+
- `saffron run` exits 2 when the files it is given hold no scenario (only
|
|
42
|
+
step sets, or nothing yet), where it said "0 passed" and exited 0.
|
|
43
|
+
- Amount checks read a minus spaced off at the start (`- €5.00`) or after
|
|
44
|
+
a currency (`CHF-5.00`) as negative, and a hyphen after another word
|
|
45
|
+
(`ID-123`) as no sign.
|
|
46
|
+
|
|
47
|
+
- Agent-blind secrets: the recording agent no longer sees `{env:...}`
|
|
48
|
+
values. It is given the token and types the token; Saffron's browser
|
|
49
|
+
tools put the value into the page, and every tool result (snapshots,
|
|
50
|
+
echoed code, console, network, `browser_evaluate`) and every Saffron
|
|
51
|
+
reply comes back with the value as its token, as do the prompt's base
|
|
52
|
+
URL and a heal prompt's replay failure and page address. That covers a
|
|
53
|
+
snapshot line Playwright single-quotes, a value Playwright's call log
|
|
54
|
+
cuts with "…", and a downloaded file quoted in a reply. A token counts
|
|
55
|
+
only where the browser types, picks, opens or uploads it (`{env:...}` in
|
|
56
|
+
`browser_evaluate` code is refused), and a word the scenario or a data
|
|
57
|
+
value it uses writes with a short value glued in (`qa{env:PIN}`) is
|
|
58
|
+
shown back as written. The
|
|
59
|
+
recording reads the browser's own results, so a value quoted in
|
|
60
|
+
Playwright's echoed code is masked to its token there too. This holds
|
|
61
|
+
for Claude, Codex, Antigravity and Cursor alike. The guide has a diagram
|
|
62
|
+
of the flow and lists what remains: values under three characters, a
|
|
63
|
+
page that shows the value transformed or only part of it, code that
|
|
64
|
+
reads a field the value was typed into, same-text matches masked along
|
|
65
|
+
with the value, and a check on the value shown in another case.
|
|
66
|
+
|
|
67
|
+
- `saffron eject` writes the committed recordings as Playwright Test specs
|
|
68
|
+
you own from then on: one spec per feature file, each Gherkin step a
|
|
69
|
+
`test.step`, a helper file beside them, and `playwright.config.ts` when
|
|
70
|
+
there is none. An exported test is never looser than replay: it may fail
|
|
71
|
+
where replay would pass, never the other way, and what has no faithful
|
|
72
|
+
form yet (amount checks, patterns or selectors holding a row value, JSON
|
|
73
|
+
response checks) is written as `test.fixme` with the reason. `{env:...}`
|
|
74
|
+
values are read when the test runs and never written into a file. The
|
|
75
|
+
exported checks read an element's text exactly as replay does, so CSS
|
|
76
|
+
`text-transform` or text split over blocks cannot pass there and fail in
|
|
77
|
+
replay. Two feature files that would share a spec file stop the command
|
|
78
|
+
before anything is written.
|
|
79
|
+
|
|
80
|
+
- UI tests can check JSON response fields by path, including nested fields
|
|
81
|
+
and array entries. Equality checks preserve string, number, boolean and
|
|
82
|
+
null types; presence checks distinguish a missing field from null.
|
|
83
|
+
Failures name the field and expected and actual values, with secrets
|
|
84
|
+
masked. These checks replay without AI and Then assertions stay strict.
|
|
85
|
+
Recordings using them require format version 5, so older runners refuse
|
|
86
|
+
them with update guidance instead of skipping the JSON checks. A field's
|
|
87
|
+
path is kept as recorded: a row value that sits in it by chance
|
|
88
|
+
(`/errors/0`) never makes another row check another field.
|
|
89
|
+
|
|
90
|
+
- Match row values in every selector-chain container, including single-quoted
|
|
91
|
+
text. Such a recording is format version 5, since 1.0.1 would fill a
|
|
92
|
+
single-quoted value in without a row boundary.
|
|
93
|
+
- A row value keeps its boundary in a pattern next to an atom that may
|
|
94
|
+
match nothing or anything (`\W*`, `\d*`, `.+`, `\S+`, `[^,]+`, `\P{L}+`),
|
|
95
|
+
a lookaround (`(?!-)`, `(?<!-)`), or a group edge or alternation
|
|
96
|
+
(`(x|<id>)`): `Order <id>.+` fails row "1" on "Order 12 shipped".
|
|
97
|
+
- An XPath selector (`xpath=//button`) replays: it was read as a regex
|
|
98
|
+
with the flags "button" and threw.
|
|
99
|
+
- An Outline with `As "<who>"` steps recorded before 1.0.1 records again
|
|
100
|
+
once on a row of another person, instead of replaying the first person's
|
|
101
|
+
values; the new recording refuses the other rows by name.
|
|
102
|
+
- Amounts: a minus spaced off at the start (`- €5.00`, `- 5,00 €`) or
|
|
103
|
+
glued after a currency (`CHF-5.00`, as Swiss formats write it) is
|
|
104
|
+
negative, and a hyphen after another word (`ID-123`) is no sign.
|
|
105
|
+
- A run of feature files with no scenario in them exits 2 and says so,
|
|
106
|
+
instead of "0 passed". A step set file that does not parse is named in
|
|
107
|
+
the error. Feature edits apply to files written with CRLF and keep their
|
|
108
|
+
line endings. A recording is written beside and renamed over, so a
|
|
109
|
+
stopped run never leaves half of one.
|
|
110
|
+
- Mask case-changed encoded secrets in capture names, reports and traces.
|
|
111
|
+
- Read labelled counts such as `Guests (2 pax)` as positive, also where
|
|
112
|
+
the label is a currency code in lower case (`Pens (2 pen)`). Accounting
|
|
113
|
+
parentheses still denote negative amounts with currency notation.
|
|
114
|
+
|
|
115
|
+
## 1.0.1: Amounts, downloads and dates as the app shows them
|
|
116
|
+
|
|
117
|
+
Released 2026-10-21.
|
|
118
|
+
|
|
119
|
+
**Upgrading from 1.0.0: some recordings can turn red.** Checks are
|
|
120
|
+
stricter, so a recording 1.0.0 passed can fail after `npm update`, with
|
|
121
|
+
nothing changed in the application. It never turns a failing check green.
|
|
122
|
+
What changed:
|
|
123
|
+
|
|
124
|
+
- A role's name matches exactly (a partial name needs an explicit name
|
|
125
|
+
regex), attribute patterns are case-sensitive, and an input-value check
|
|
126
|
+
needs a visible input.
|
|
127
|
+
- A text or input-value check records an explicit expected value. Expected
|
|
128
|
+
text, a text pattern, a download check, an expected URL, or a target's
|
|
129
|
+
text, label or placeholder that a blank Examples or data value leaves
|
|
130
|
+
partly empty fails instead of matching more than it means.
|
|
131
|
+
- A response check counts only requests started since the latest browser
|
|
132
|
+
action, so an earlier successful save no longer satisfies it.
|
|
133
|
+
- An Outline recording that cannot follow another row (short values,
|
|
134
|
+
dates without a year, a role taken from a row, an unscoped action,
|
|
135
|
+
check or capture in a section a column names, a value typed in a
|
|
136
|
+
column's browser session, a step naming a column that recorded no action
|
|
137
|
+
writing it, a state check naming a column whose value it does not show,
|
|
138
|
+
as in `Then <user> sees the dashboard`) refuses that row with guidance
|
|
139
|
+
instead of passing on row 1's data. Login Outlines of that shape now
|
|
140
|
+
pass only on the recorded row: write the user in the text the check
|
|
141
|
+
expects (`Then the dashboard greets "<user>"`) and record again.
|
|
142
|
+
|
|
143
|
+
Re-record what fails (`saffron run --rerecord <file>:<line>`) or correct
|
|
144
|
+
the step the message names. Recordings 1.0.1 writes with its new features
|
|
145
|
+
(localized dates, several downloads per click, the new row bindings) are
|
|
146
|
+
refused by 1.0.0 by name, so a team updates together.
|
|
147
|
+
|
|
148
|
+
Agent plugins
|
|
149
|
+
|
|
150
|
+
- Declare documentation, privacy, license terms and support links for the
|
|
151
|
+
Claude directory listing, with explicit README links as a fallback.
|
|
152
|
+
|
|
153
|
+
- Prepare the agent plugin listings with the Saffron icon and documented
|
|
154
|
+
data handling. The Codex listing adds a shorter subtitle, local
|
|
155
|
+
execution requirements, prerequisites and links to support, data handling
|
|
156
|
+
and the software license.
|
|
157
|
+
|
|
158
|
+
Runner
|
|
159
|
+
|
|
160
|
+
- A row refused because its check only shows that something is visible or
|
|
161
|
+
in a state (`Then <user> sees the dashboard`) now says so, and suggests
|
|
162
|
+
writing the value in the text the check expects
|
|
163
|
+
(`Then the dashboard greets "<user>"`).
|
|
164
|
+
- A fallback selector's text (`:has-text("Order <id>")`, `text=Order
|
|
165
|
+
<id>`) finds row 1 only as a whole number or word, not inside "Order
|
|
166
|
+
12", and a blank row value at its edge is refused, as in other checks.
|
|
167
|
+
- A pattern that joins letters or digits to a row value itself
|
|
168
|
+
(`^<prefix>\d+$`) keeps them joined at replay.
|
|
169
|
+
- A secret shown lower-cased and percent-encoded in a URL is masked, and
|
|
170
|
+
so is a number secret shown with spaces or dashes between its digits.
|
|
171
|
+
- A day a step writes as plain `{date+1}` and the app shows in its own
|
|
172
|
+
format (`21/10/2026`, `Oct 21, 2026`) is recorded as the token in that
|
|
173
|
+
format, so the check moves with the calendar.
|
|
174
|
+
- A quoted CSV field keeps its own spaces. A double-quoted `.env` value
|
|
175
|
+
takes `\"` and `\\` escapes. A count in parentheses ("Cart (3)") is no
|
|
176
|
+
longer read as a negative amount.
|
|
177
|
+
|
|
178
|
+
- Mask mixed-case percent escapes in capture names, reports and traces
|
|
179
|
+
through the shared secret masker.
|
|
180
|
+
- Keep state checks bound to their recorded row even when the assertion
|
|
181
|
+
step also presses a key or performs another browser action.
|
|
182
|
+
- Match row values with Unicode-aware regex boundaries, including selector
|
|
183
|
+
fallbacks and downloads. Non-ASCII suffixes cannot satisfy a shorter value.
|
|
184
|
+
- Refuse tokens inside regex repeat counts instead of silently changing
|
|
185
|
+
the assertion. Use a separate Scenario with a literal repeat count.
|
|
186
|
+
|
|
187
|
+
- Mask URL-encoded secrets with lowercase or mixed-case percent escapes,
|
|
188
|
+
including escaped patterns, before saving recordings.
|
|
189
|
+
- Bind state checks to their recorded Outline row when the check does not
|
|
190
|
+
follow its column. A column at the start of the sentence no longer
|
|
191
|
+
bypasses this safeguard.
|
|
192
|
+
- Apply row-value boundaries to selector regex fallbacks and download
|
|
193
|
+
checks. Preserve character-class syntax when resolving regex tokens.
|
|
194
|
+
- Classify Given and inherited setup steps using the feature's language,
|
|
195
|
+
including Latvian keywords shared with other languages.
|
|
196
|
+
|
|
197
|
+
- Rendered-text capture tests allow time for capture stability and browser
|
|
198
|
+
round trips on CI. Hidden and empty text must still fail within the same
|
|
199
|
+
short test timeout.
|
|
200
|
+
- Runner CI installs WebKit alongside Chromium and Firefox so the existing
|
|
201
|
+
tests for all three browser engines can run on GitHub Actions.
|
|
202
|
+
|
|
203
|
+
- Review fixes: a provider or agent error is masked for secrets in full
|
|
204
|
+
before it is cut to its first line, so a secret spanning lines no
|
|
205
|
+
longer shows in the console or a later scenario's error. A typed value
|
|
206
|
+
submitted with Enter presses it on the field typed into, not whatever
|
|
207
|
+
has focus after an input handler ran. A date token with a locale tag
|
|
208
|
+
Node.js rejects (`en-x`) is marked in the editor instead of stopping the
|
|
209
|
+
diagnostics. A report row's `ranAt` is validated like the run's times,
|
|
210
|
+
and the HTML page is rendered before either report file is replaced.
|
|
211
|
+
A quoted value containing a word like "clear" no longer hides a lost
|
|
212
|
+
typed value. An amount with its minus before the currency (`-$5.00`)
|
|
213
|
+
reads as negative, so doubling it is never checked against $10.00. A
|
|
214
|
+
capture named like a token (`price {date}`) keeps its name in an amount
|
|
215
|
+
check at replay. A click that starts several downloads stores the count
|
|
216
|
+
as a number of its own, which no Outline pass rewrites, and the
|
|
217
|
+
recording is written as version 4, so an earlier runner refuses it
|
|
218
|
+
instead of checking the first file. Recording judges a download check
|
|
219
|
+
by replay's blank rule and resolves `{unique:...}` inside Examples
|
|
220
|
+
values, as replay does.
|
|
221
|
+
|
|
222
|
+
- Fourth review round. Outlines: a step that names a column but recorded
|
|
223
|
+
nothing that writes it (the page already showed "myself", or a box was
|
|
224
|
+
ticked) binds the recording to its row, so the "someone else" row is
|
|
225
|
+
refused instead of replaying row 1's page; a recording made before this
|
|
226
|
+
is recorded again once on a row that differs, and the agent is asked to
|
|
227
|
+
pick the row's value even where the page already shows it. A blank
|
|
228
|
+
first row no longer refuses every filled row: a filled row records once
|
|
229
|
+
more, and that recording serves the blank rows. A typed value equal to
|
|
230
|
+
the section's name no longer passes for scoping where the step says it
|
|
231
|
+
is a section (`in the "Double" room card`); captures and an assertion
|
|
232
|
+
step's checks must name the section too, and `the country selector in
|
|
233
|
+
"<section>"` and a hyphenated container word (`time-selection dialog`)
|
|
234
|
+
read as sections. A dialog's button the scoped action opened, scoped to
|
|
235
|
+
the dialog, and a wait, need not name the section. Literal braces in a
|
|
236
|
+
typed value (a JSON address) are text in a person's browser session, and
|
|
237
|
+
a short `{data:...}` value the step writes is the step's own. A date cell
|
|
238
|
+
written with its month's name (`{date+1:MMM D}`) follows the row in any
|
|
239
|
+
step, instead of being recorded again every day. Recording: performing
|
|
240
|
+
several flagged actions again replaces each in its own place, one by
|
|
241
|
+
one, and one not performed again stays where it was. A name Playwright's
|
|
242
|
+
MCP shortened (`'26. Oktober'` for "26. Oktober 2026", `'Room'` for
|
|
243
|
+
"Room 101") is recorded in full from the snapshot the agent clicked in,
|
|
244
|
+
as replay matches names exactly. Through the trace bridge, no page
|
|
245
|
+
fingerprint is taken before a browser action, which had made every
|
|
246
|
+
click by snapshot ref fail. Replay: a captured value is taken once the
|
|
247
|
+
page has stopped changing it for half a second (sooner for a short
|
|
248
|
+
timeout), so a total drawn again after the last change is not read
|
|
249
|
+
stale. Proposals: a proposal git checked out with CRLF line ends is
|
|
250
|
+
still the one this machine filed. Review: a proposal whose only change
|
|
251
|
+
is beside the steps (a row binding, the row it was recorded on, a data
|
|
252
|
+
value) says so in `saffron diff` and the report. `saffron status` judges
|
|
253
|
+
an Outline row the recording refuses as refused, not stale. A CI run's
|
|
254
|
+
results say when the run was interrupted. Step reuse records afresh
|
|
255
|
+
instead of keeping a capture under another value's name, and a fix
|
|
256
|
+
carried to other scenarios never copies a `<column>` into one without
|
|
257
|
+
it. The editor checks date tokens in doc strings too, never on a
|
|
258
|
+
comment line.
|
|
259
|
+
|
|
260
|
+
- Fifth review round. A step naming a column is judged by its own
|
|
261
|
+
actions: another step writing the column, or another column sharing its
|
|
262
|
+
value, no longer lets an empty step pass on other rows. Step reuse never
|
|
263
|
+
puts such a step back for a row whose value differs, so the row records
|
|
264
|
+
it afresh. A check on a bare role (`the error message in "<language>"`)
|
|
265
|
+
takes no section from the quote, and a wait in a section step never
|
|
266
|
+
binds the row. `saffron status` marks a refused row with ✗ and says why,
|
|
267
|
+
instead of calling its recording unreadable and counting it as replaying.
|
|
268
|
+
|
|
269
|
+
- Sixth review round. Recording: a full button name is read only from the
|
|
270
|
+
page snapshot a browser call shows, never from a response body or the
|
|
271
|
+
console. Corrections a step performs again, one by one or in one call,
|
|
272
|
+
each take the place of the flagged action of their kind and element, so
|
|
273
|
+
a fill stays before the Save after it; one that reopens a dialog
|
|
274
|
+
replaces the click that first opened it instead of clicking twice. A
|
|
275
|
+
check spelling its own value (`Then ... "Guest: Tor"`) keeps it, and no
|
|
276
|
+
longer reads another step's table cell. A value typed from
|
|
277
|
+
`{data:users.<who>.email}` is recorded as that reference, so an edited
|
|
278
|
+
data file reaches replay. `the "<room>" room card should show ...` is a
|
|
279
|
+
section without "in". Replay: a capture waits for the page's fetch and
|
|
280
|
+
XHR requests to be answered (up to 3 seconds) before reading.
|
|
281
|
+
|
|
282
|
+
- Seventh review round. One value two references in a step read
|
|
283
|
+
(shipping and billing both "Home") stays as typed and binds the row,
|
|
284
|
+
instead of becoming one of them; in different steps, each keeps its own
|
|
285
|
+
reference. An alert or other bare-role check inside a section the step
|
|
286
|
+
names explicitly must be scoped there. Step reuse keeps out a `When`
|
|
287
|
+
step that checks something but sets nothing of its column. A dialog
|
|
288
|
+
reopened while correcting a step replaces the click that first opened
|
|
289
|
+
it, across the waits between them, and a wait performed again is filed
|
|
290
|
+
once. Only the very same action counts as reopening (the same target in
|
|
291
|
+
the same scope, with the same click options): a click focusing a field
|
|
292
|
+
never replaces its fill, nor a click on a checkbox of the same name. A
|
|
293
|
+
short shared value ("NY"), a blank one, or one written escaped in a
|
|
294
|
+
pattern or selector, binds the row like any other. A `{data:...<column>...}`
|
|
295
|
+
value blank on the recorded row records once more on a filled row,
|
|
296
|
+
as a blank cell does, and one kept in another spelling (`ship-home`)
|
|
297
|
+
binds the row. A recording made before these bindings that still holds
|
|
298
|
+
such a value literally records once more on a row reading another. A
|
|
299
|
+
later check still holding a value another step's reference read (a
|
|
300
|
+
summary showing "Home") binds the row too, and a step that reads a
|
|
301
|
+
value and also spells it as its own constant (`Shipping "{data:...}"
|
|
302
|
+
and Billing "Home"`) binds the row, as which field was which cannot be
|
|
303
|
+
told. A short value ("M", "NY") counts only as a whole word, never
|
|
304
|
+
inside "My cart", and one inside the step's own text ("Shipping to NY")
|
|
305
|
+
becomes its reference. A recording made before this that typed the
|
|
306
|
+
reference for the step's own constant records once more. A value the
|
|
307
|
+
step spells inside a longer word ("Johnson" beside a first name
|
|
308
|
+
"John") binds the row too, and step reuse never puts back a check still
|
|
309
|
+
holding a value a reference read on a row that differs.
|
|
310
|
+
|
|
311
|
+
- Twelfth review round. Recording: a data value never rewrites a
|
|
312
|
+
`<column>` placeholder or a `{data:...}` reference's path ("email" in
|
|
313
|
+
`<email>`), and a tag in a selector (`button`) is never data. A value a
|
|
314
|
+
step spells itself stays its own when another step reads it ("Johnson"
|
|
315
|
+
beside a first name "John"). A check naming a column nothing in the
|
|
316
|
+
recording follows (`Then the banner is <state>`, recorded as "not
|
|
317
|
+
visible") binds the row. A role, an amount written another way
|
|
318
|
+
("12,50" checked as 12.50) or another spelling (`address-home-street`)
|
|
319
|
+
of a `{data:...<column>...}` value binds the row too, and a password
|
|
320
|
+
read through `{data:users.<who>.password}` from `{env:...}` follows the
|
|
321
|
+
row instead of keeping the first row's variable. Step reuse: a `When`
|
|
322
|
+
recording never seeds a `Then` of the same words, and a check holding a
|
|
323
|
+
value a joined key (`zone-<s>`) read, or expecting nothing for a blank
|
|
324
|
+
one, is never put back, also from a recording that names no row.
|
|
325
|
+
Replay: a `{date...}` inside a data value resolves to its day, as it
|
|
326
|
+
did while recording. The review that followed: secrets are masked
|
|
327
|
+
before any other pass, so a row, table or data value inside one never
|
|
328
|
+
cuts it up and leaves the rest in the recording. A state check ("not
|
|
329
|
+
visible", "disabled") naming a column binds the row even where another
|
|
330
|
+
step sets that state, and so does a check showing a date column's day
|
|
331
|
+
in the page's own format; a check stating its own text may name a
|
|
332
|
+
column to describe the row. Another step's value in an action step
|
|
333
|
+
("Ship to NY") binds the row, a session's column (`As "<who>"`) is
|
|
334
|
+
never cut out of a longer word ("Hannah"), a data value holding a
|
|
335
|
+
formatted date becomes its reference, and a constant spelled inside a
|
|
336
|
+
longer word stays however often the step names a reference. A changed
|
|
337
|
+
data file is noticed where a check holds the old value as a role or an
|
|
338
|
+
amount. Step reuse refuses joined keys without a separator
|
|
339
|
+
(`zone<s>`) and empty checks in any spelling (`^(?:)$`), and a
|
|
340
|
+
recording whose data holds a date is version 3, which saffron-ai 1.0.0
|
|
341
|
+
refuses rather than typing `{date+1}`.
|
|
342
|
+
|
|
343
|
+
- Thirteenth review round. Outlines: a row value inside a run of letters
|
|
344
|
+
("red" in "Delivered", "Pro" in "Products") no longer binds the row; a
|
|
345
|
+
camelCase part ("SelectedRed") still does. Rows whose data reads blank
|
|
346
|
+
on every row no longer record each other forever. A row value holding a
|
|
347
|
+
secret's value ("admin@shop.test" beside a variable "admin") is the
|
|
348
|
+
row's `<column>`, and a secret equal to a row's lookup key leaves the
|
|
349
|
+
step's `{data:...}` reference alone. A state check ("Promo" hidden, an
|
|
350
|
+
attribute's pattern) binds the row whatever its element is named. A
|
|
351
|
+
dialog answered after announcing the next step is armed before its
|
|
352
|
+
trigger, in the trigger's step. The fingerprint a row binding keeps of
|
|
353
|
+
a secret is salted and slow, so a short one is not read back from the
|
|
354
|
+
recording (version 3); older plain ones still match. A blank Examples
|
|
355
|
+
cell beside a step with something to check is version 3, and step reuse
|
|
356
|
+
refuses a one-character joined key (`zone0`). An older recording is
|
|
357
|
+
recorded again once a data value it reads holds a date. Replay: a row
|
|
358
|
+
value starting or ending an expected text, URL, response URL or text
|
|
359
|
+
target is never found inside a longer word or number ("Order 1" never
|
|
360
|
+
passes on "Order 12"). A pending proposal that fails to replay does so
|
|
361
|
+
in a browser of its own, so the agent records in a clean one. CLI:
|
|
362
|
+
`saffron prune --yes` names what it could not delete and exits 1; a
|
|
363
|
+
hook killed by a signal says so; `.env` reads `export` lines and drops
|
|
364
|
+
inline comments; "$.50" is 0.50. The language server runs a scenario
|
|
365
|
+
sharing its name with another by line, reads `SAFFRON_ENV` from the
|
|
366
|
+
project's `.env`, and checks date tokens only in steps, tables and doc
|
|
367
|
+
strings. Then: a check that an element found by its text is absent
|
|
368
|
+
("Promo" not visible) binds the row like any state check; a role with
|
|
369
|
+
its text never finds a row value inside a longer one either; re-masking
|
|
370
|
+
a secret never cuts into a `<column>` it sits in ("admin" in
|
|
371
|
+
`<adminEmail>`); and "-$.50" and "($.50)" are negative. The review after
|
|
372
|
+
that: a secret percent-encoded in a recorded URL (`admin%40shop.test`)
|
|
373
|
+
is masked too, and a URL check also reads the page's URL decoded;
|
|
374
|
+
another step's data value shown in another case or as a slug ("MUG")
|
|
375
|
+
binds the row; two columns holding one `{env:...}` password record once
|
|
376
|
+
more for a row typing them apart instead of being refused for good; a
|
|
377
|
+
column opening a Then step that another step writes (`Then <user> sees
|
|
378
|
+
the dashboard`) describes the row; a three-letter data value inside
|
|
379
|
+
text ("1 x Mug") follows its reference. A response URL pattern bounds a
|
|
380
|
+
row value wherever it stands, `saffron status` judges every Examples row
|
|
381
|
+
and finds an install in a parent folder, a pending proposal failing
|
|
382
|
+
for another row keeps its screenshot and trace, a quoted `.env` value
|
|
383
|
+
followed by a comment loses its quotes, and an empty Given in any
|
|
384
|
+
language is not sent back. The full review after that: a secret
|
|
385
|
+
percent-encoded in a pattern, written in another case, or in an
|
|
386
|
+
`encodeURI` URL is masked; a data value that is a variable follows its
|
|
387
|
+
reference inside text too; an amount a text check shows in another
|
|
388
|
+
format, a state check found by its text (visible or not), and a number
|
|
389
|
+
only inside a longer one are each judged as they should be; a column
|
|
390
|
+
opening a check is its subject only when another step writes it and the
|
|
391
|
+
check does not state it (`<state> is ...`); a multi-word value in a URL
|
|
392
|
+
(`q=red+shoes`) follows the row. Replay bounds a row value in every
|
|
393
|
+
pattern (text, name, attribute, response URL and body) and refuses a
|
|
394
|
+
blank one left inside an attribute or body pattern, reads response URLs
|
|
395
|
+
decoded, and keeps a regex's own named groups. A fresh recording that
|
|
396
|
+
never announced its last steps is sent back instead of filed as
|
|
397
|
+
verified. A quoted CSV field after a space is read whole, and a row with
|
|
398
|
+
more fields than its header is refused. `.env`: the last of a repeated
|
|
399
|
+
key wins, quotes match non-greedily, and inherited names are ordinary
|
|
400
|
+
keys; neither data nor rows read inherited properties. `saffron status`
|
|
401
|
+
counts unreadable recordings apart from stale ones, `saffron trace`
|
|
402
|
+
prints report names without terminal escapes, and a run without the
|
|
403
|
+
agent says so. A click performed twice on purpose while correcting a
|
|
404
|
+
step is kept twice; a reopened download trigger takes its download wait
|
|
405
|
+
with it, so the count is not doubled; and an armed action goes before
|
|
406
|
+
its trigger even when the same call waited for something after it.
|
|
407
|
+
|
|
408
|
+
- An Outline whose browser session comes from a column (`As "<who>" ...`)
|
|
409
|
+
no longer lets another row's person type the recorded person's value: a
|
|
410
|
+
step in that session typing a value the step does not write binds the
|
|
411
|
+
recording to its row (written as version 3), so the other rows are
|
|
412
|
+
refused with guidance instead of passing on the first person's data.
|
|
413
|
+
|
|
414
|
+
- Third review round. Recording: each action and check in a section a
|
|
415
|
+
column names must be scoped to it (one scoped action no longer covers an
|
|
416
|
+
unscoped fill beside it), a quoted value the step types is no section,
|
|
417
|
+
and performing a flagged action again replaces only that action, in its
|
|
418
|
+
place, with its download wait. A download is counted from the browser's
|
|
419
|
+
own events only, never twice. An amount the step writes itself stays the
|
|
420
|
+
step's, a column two written amounts match stays literal, a date cell's
|
|
421
|
+
day stays literal only in the steps that name its column, tokens that
|
|
422
|
+
always read alike (`{date+7:dddd}`, `{date+14:dddd}`) are one, and "an
|
|
423
|
+
empty" field reads as nothing to type. Format: a recording made before
|
|
424
|
+
a new-syntax date token was added to its scenario is recorded again
|
|
425
|
+
once as version 3, a "container" row binding is version 3, and
|
|
426
|
+
`rowsUseNewDates` is part of what a proposal binds. Replay: a download
|
|
427
|
+
check, an expected URL, or a target's text, label or placeholder that a
|
|
428
|
+
blank row value leaves partly empty fails instead of matching more; a
|
|
429
|
+
newer dialog answer replaces one no dialog used; a targeted upload that
|
|
430
|
+
starts a download is waited for; a retry sets the chosen files again;
|
|
431
|
+
an amount check's quoted names are never resolved. Runs: `--rerecord`
|
|
432
|
+
refuses, deleting nothing, when no browser of the run can record or the
|
|
433
|
+
agent cannot start; an offset time zone is refused; "retry in 41.8s"
|
|
434
|
+
is a passing throttle; Claude overage use is no limit.
|
|
435
|
+
|
|
436
|
+
- Second review round. Recording: a section a column names that the
|
|
437
|
+
recording ignores binds the row, so another row is refused rather than
|
|
438
|
+
acting in row 1's section; a place or manner (`in "Oslo"`) is no longer
|
|
439
|
+
taken for a section, and a `Given` the page already satisfies is not
|
|
440
|
+
asked for an action. Amounts: accounting, trailing and en-dash negatives,
|
|
441
|
+
decimals such as `0.125`, and an amount written in another format than its
|
|
442
|
+
Examples cell follow the row; a misplaced quote in an expression is
|
|
443
|
+
refused. Dates: a day the recorded row's cell holds stays literal, two
|
|
444
|
+
tokens that read alike stay literal, only whole dates are made tokens
|
|
445
|
+
outside their own step, and an Outline whose other rows use the new date
|
|
446
|
+
syntax is written as version 3. `{date-of-birth}` is typed as written,
|
|
447
|
+
offsets past a hundred years and locale-like typos are named, and a time
|
|
448
|
+
zone must be spelled as browsers spell it. Downloads: one started in a
|
|
449
|
+
popup tab or after its click returned is credited to that click, one
|
|
450
|
+
reported by a later result raises the count, opening a file's URL and a
|
|
451
|
+
chosen file that starts a download replay, and a response body is never
|
|
452
|
+
read as actions. Runs: a throttle that clears in seconds does not stop
|
|
453
|
+
AI sessions, "Quota exceeded" and a Claude subscription limit do, and
|
|
454
|
+
Ctrl-C before any scenario finished keeps the last report.
|
|
455
|
+
|
|
456
|
+
- **Amounts that add up.** A step such as `Then the basket total should be
|
|
457
|
+
"before" plus "breakfast"` records an `expectAmount` check: the target
|
|
458
|
+
must show the amount an expression over captured values states
|
|
459
|
+
(`"before" + "breakfast"`, `"total" - "discount"`, `"night" * 2`),
|
|
460
|
+
compared to the cent. Amounts are read in either decimal style
|
|
461
|
+
(`NOK 1 234,50`, `$1,234.56`, `CHF 1'234.50`); an element showing two
|
|
462
|
+
numbers is refused rather than guessed. A total that changed by the
|
|
463
|
+
wrong amount now fails, where a "differs" check passed. The agent
|
|
464
|
+
records it with `saffron_assert` kind `amount`, which refuses a name no
|
|
465
|
+
step captures. Capture names in the expression are identifiers: no
|
|
466
|
+
Outline, data, date or secret pass rewrites them, and a secret-bearing
|
|
467
|
+
one is renamed together with its capture.
|
|
468
|
+
|
|
469
|
+
- Agents are told that other scenarios reuse only accepted recordings:
|
|
470
|
+
the run's "proposals pending" line, the `project_status` MCP tool and
|
|
471
|
+
the bundled skill say to accept a shared setup before recording more
|
|
472
|
+
scenarios that use it, instead of paying for it in every one. The skill
|
|
473
|
+
also asks for a mid-scenario `Then` at each checkpoint of a route (a 3DS
|
|
474
|
+
challenge, a redirect), the exact control a variant uses, and the
|
|
475
|
+
section a step acts in when labels repeat.
|
|
476
|
+
|
|
477
|
+
- Fixes from a field report on recording a checkout suite with Codex.
|
|
478
|
+
`saffron_done` refuses a pass once while an action step recorded no
|
|
479
|
+
browser action: announcing a step performs nothing, and a step's action
|
|
480
|
+
can land under the next step. A step that acts in a named container
|
|
481
|
+
(`in "Your contact information"`) with a target that ignores it is sent
|
|
482
|
+
back too, since replay acts on the first match; performing it again
|
|
483
|
+
replaces the unscoped action. The step reply and the agent's
|
|
484
|
+
instructions say that announcing performs nothing, ask readiness steps
|
|
485
|
+
to record the condition they wait for, and ask for a fresh snapshot when
|
|
486
|
+
a ref no longer resolves. A second pass stands, so a step that needs
|
|
487
|
+
nothing still finishes.
|
|
488
|
+
|
|
489
|
+
- The lost-value check no longer flags a step that clears a field or
|
|
490
|
+
passes `""`: a quoted field name is not a typed value. An empty fill on a
|
|
491
|
+
step that types a value is still flagged. A blank first Examples row
|
|
492
|
+
binds to `<column>`, and an expected message held in a row's
|
|
493
|
+
`{data:...}` token follows each row.
|
|
494
|
+
|
|
495
|
+
- `--rerecord` records afresh over a pending proposal from an earlier
|
|
496
|
+
run, not only a row-bound one, instead of replaying it at zero AI. Rows
|
|
497
|
+
of the same run share the new recording.
|
|
498
|
+
|
|
499
|
+
- When an agent session fails on its provider's usage limit, no further
|
|
500
|
+
agent session starts in that run, on any worker (the agent describing
|
|
501
|
+
such a message in the application does not count, and the message is
|
|
502
|
+
stored with the scenario's secrets masked): the scenarios that need one
|
|
503
|
+
fail at once with the provider's message, and replays carry on. A
|
|
504
|
+
session in which the agent made no tool call at all says its tools did
|
|
505
|
+
not reach it.
|
|
506
|
+
|
|
507
|
+
- The run summary counts pending proposals whose proof replay failed
|
|
508
|
+
apart from the others, and the report splits them into proof passed,
|
|
509
|
+
proof failed and not proved. A run of named scenarios marks each report
|
|
510
|
+
row kept from an earlier run with when that run started; shards of one
|
|
511
|
+
run are not marked. A pending proposal that does not replay for another Examples
|
|
512
|
+
row names the columns that row types differently, and which of them the
|
|
513
|
+
failing step uses, instead of guessing at the cause.
|
|
514
|
+
|
|
515
|
+
- Typing with submission checks Enter in its own response window, so a
|
|
516
|
+
successful autosave cannot hide a failed submission during healing.
|
|
517
|
+
- Completed form fields and typing stay in the recording if a later field
|
|
518
|
+
or submission fails, including when the agent retries only the failed part.
|
|
519
|
+
Compound calls are validated before their first edit.
|
|
520
|
+
- Agent shutdown allows five seconds for started calls to finish recording,
|
|
521
|
+
then cancels stuck work and fails the incomplete recording with a clear
|
|
522
|
+
timeout. Queued calls stay blocked and provider usage is retained.
|
|
523
|
+
|
|
524
|
+
- Healing checks each field of a multi-field form against its own response
|
|
525
|
+
window, matching replay. The full batch is validated before any field
|
|
526
|
+
changes. Capture tracking preserves named regex groups when step parameters
|
|
527
|
+
change and prefers a unique test ID over app-generated DOM IDs for remounts.
|
|
528
|
+
|
|
529
|
+
- Healing response assertions follow the latest agent interaction in their
|
|
530
|
+
browser session, so an earlier successful retry cannot hide a later failure.
|
|
531
|
+
Agent shutdown waits for started browser calls to finish recording before
|
|
532
|
+
closing their clients or finalizing the failed session; queued calls stay
|
|
533
|
+
blocked.
|
|
534
|
+
|
|
535
|
+
- Codex recording and healing enable the code-mode tool host while retaining
|
|
536
|
+
Saffron's isolated browser tools. Tool-call limits are enforced at the
|
|
537
|
+
gateway, including calls nested in code-mode batches, before an excess
|
|
538
|
+
call can reach the browser. No dependency patch is needed.
|
|
539
|
+
|
|
540
|
+
- Text patterns reject partly blank row, data and environment values, including
|
|
541
|
+
nested tokens. The healing assertion guard retains captured element and
|
|
542
|
+
frame identity, so another frame cannot stand in for a removed target.
|
|
543
|
+
|
|
544
|
+
- Follow-up assertion fixes: redirects retain the original request's action
|
|
545
|
+
window, and partly blank text is refused even when the blank comes through
|
|
546
|
+
an Examples cell or data file. Live comparisons retain the captured target
|
|
547
|
+
and frame when that target becomes hidden or disappears. A unique ID or
|
|
548
|
+
test ID allows ordinary DOM replacement in the same frame; duplicate
|
|
549
|
+
matches remain tied to the original element.
|
|
550
|
+
|
|
551
|
+
- Response assertions check requests started since the latest browser action,
|
|
552
|
+
so an earlier successful save cannot hide a later failed save, even when
|
|
553
|
+
the earlier response arrives late. Waits and consecutive assertions keep
|
|
554
|
+
that window open. Check a request before the next interaction or navigation.
|
|
555
|
+
- Assertions use exact role names and case-sensitive attribute patterns;
|
|
556
|
+
input value checks require a visible input. Use an explicit name regex
|
|
557
|
+
for a partial role name. Text expectations containing a blank row or data
|
|
558
|
+
value alongside other text now fail with guidance. Recording a text or
|
|
559
|
+
input-value assertion requires an explicit expected value, including `""`
|
|
560
|
+
for an intentional empty check. Existing recordings relying on these
|
|
561
|
+
looser checks must be corrected or re-recorded.
|
|
562
|
+
|
|
563
|
+
- Outline recordings refuse other rows when they retain short values in
|
|
564
|
+
action steps, dates without a year or glued to digits, an unparameterized
|
|
565
|
+
blank cell, or an element role taken from the recorded row. A whole
|
|
566
|
+
`{env:...}` or `{data:...}` field equal to a row's value is treated as
|
|
567
|
+
ambiguous even when the step names that token. These cases fail with
|
|
568
|
+
guidance instead of passing on another row's data or repeatedly recording.
|
|
569
|
+
- Step reuse refuses changes to quoted arguments containing several
|
|
570
|
+
placeholders, where it cannot safely separate their values. Row comparisons
|
|
571
|
+
keep date tokens inside environment values literal, matching replay.
|
|
572
|
+
|
|
573
|
+
- Setup hooks can create or refresh files used by `{data:...}` steps. Data
|
|
574
|
+
and referenced environment variables are validated after setup, and replay
|
|
575
|
+
reads the refreshed values. Invalid data stops the run before a browser
|
|
576
|
+
opens and still runs teardown.
|
|
577
|
+
|
|
578
|
+
- Ctrl-C during `saffron run` writes the report of what finished, with or
|
|
579
|
+
without setup and teardown configured, instead of leaving the last run's
|
|
580
|
+
report in place. The report has `"interrupted": true` and `notRun`, the
|
|
581
|
+
scenarios the run selected that did not finish, which count as neither
|
|
582
|
+
passed nor failed; the HTML report, the run's output and `saffron status`
|
|
583
|
+
say it is partial, and its history line is marked partial. The browsers
|
|
584
|
+
close and teardown runs first, then Saffron exits with code 130.
|
|
585
|
+
Playwright's own Ctrl-C handler is off, so it no longer exits before
|
|
586
|
+
teardown has run. `saffron report --merge` refuses an interrupted shard,
|
|
587
|
+
and a run of named scenarios replaces an interrupted report whole.
|
|
588
|
+
|
|
589
|
+
- The run's first line and the report's `provenance` say what produced
|
|
590
|
+
it: the provider, each model asked for and whether it came from
|
|
591
|
+
`--model`, `--heal-model` or saffron.config.json, and the installed
|
|
592
|
+
version of the vendor package (`@openai/codex-sdk`,
|
|
593
|
+
`@anthropic-ai/claude-agent-sdk`, `@cursor/sdk`) beside saffron-ai's. A
|
|
594
|
+
model nobody asked for is left unnamed; the models that answered stay in
|
|
595
|
+
`totals.models`.
|
|
596
|
+
|
|
597
|
+
- **Dates as the app shows them.** A `{date...}` format writes month and
|
|
598
|
+
weekday names (`MMMM`, `MMM`, `dddd`, `ddd`) and two-digit years
|
|
599
|
+
(`YY`), in the locale the token names (`{date+7:D. MMM:de-DE}` is "24.
|
|
600
|
+
Okt.") or in `dateLocale` from `saffron.config.json` (en-US when
|
|
601
|
+
absent). A range is one token, `{date+7 - +9:MMM D, YYYY}`: what is
|
|
602
|
+
written between the offsets is written between the days, and what both
|
|
603
|
+
share is written once ("Oct 24 - 26, 2026", "Oct 30 - Nov 1, 2026",
|
|
604
|
+
"Dec 31, 2026 - Jan 2, 2027"). A suite no longer needs a setup script to
|
|
605
|
+
compute the stay it picks and the range it checks.
|
|
606
|
+
|
|
607
|
+
- **One clock.** `timezone` in `saffron.config.json` sets the calendar
|
|
608
|
+
date tokens count days on and the time zone every browser runs in, so
|
|
609
|
+
the page and the tokens agree on today, around midnight and through
|
|
610
|
+
clock changes. Every token of one replayed scenario resolves on the
|
|
611
|
+
clock the scenario started with.
|
|
612
|
+
|
|
613
|
+
- The agent is given each date of a scenario resolved, on the clock its
|
|
614
|
+
recording started with, instead of the token; the recording turns each
|
|
615
|
+
of those days back into its token, so a date written in a step's text in
|
|
616
|
+
any format moves with the calendar, not only an ISO one.
|
|
617
|
+
|
|
618
|
+
- A `{date...}` token Saffron cannot read, or with a locale this Node.js
|
|
619
|
+
does not know, stops the run before any browser opens, naming it and the
|
|
620
|
+
scenario; the language server marks it in `.feature` and `.saffron`
|
|
621
|
+
files. The TextMate grammar colours ranges.
|
|
622
|
+
|
|
623
|
+
- A recording with a range, a locale, a name or a two-digit year in a date
|
|
624
|
+
token is written as version 3 of the recording format, so saffron-ai
|
|
625
|
+
1.0.0, which would resolve it wrong, refuses it by name.
|
|
626
|
+
|
|
627
|
+
- **What a download says.** A booking confirmation's "Add to Apple
|
|
628
|
+
Calendar" link could only be checked by its `href` and file name, never
|
|
629
|
+
by the dates and identifiers inside the file. Now the click that
|
|
630
|
+
downloads a file records with the download armed before it
|
|
631
|
+
(`waitForDownload`, as a dialog is), and replay waits for that
|
|
632
|
+
download, fails the click when none starts, and keeps the file in
|
|
633
|
+
`.saffron/artifacts/<feature>/<scenario>/downloads/`. A `Then` step
|
|
634
|
+
such as `Then the calendar file should contain
|
|
635
|
+
"DTSTART;VALUE=DATE:{date+7:YYYYMMDD}"` records a content check
|
|
636
|
+
(`expectDownload`): text the scenario's latest download contains, a
|
|
637
|
+
regular expression it matches (case-sensitive, `^` and `$` at each
|
|
638
|
+
line), or both, with `{date...}`, `{data:...}`, `{env:...}` and
|
|
639
|
+
Examples values resolved.
|
|
640
|
+
Line breaks read as `\n`, a calendar or contact file's folded lines
|
|
641
|
+
read whole, and only UTF-8 text files up to 5 MB are checked. The check
|
|
642
|
+
is never healed. While recording, Saffron checks it against the file
|
|
643
|
+
the agent downloaded (`saffron_assert` kind `download`) and shows the
|
|
644
|
+
agent the start of the file when it does not hold. A click that
|
|
645
|
+
downloads a file is no longer lost from a recording made without
|
|
646
|
+
tracing: Playwright's MCP, attached over CDP, could not copy the file
|
|
647
|
+
and reported the click as failed, so nothing was recorded. Saffron now
|
|
648
|
+
keeps the file through its own connection and drops that one error.
|
|
649
|
+
A large page no longer hides a download from the recorder either: a
|
|
650
|
+
tool result cut to fit the agent's context keeps the sections the
|
|
651
|
+
recorder reads (its code, tabs and Events) whole, even where the cut
|
|
652
|
+
would split one, and the parts of a compound call share one allowance.
|
|
653
|
+
A click that starts several downloads records how many; replay waits for
|
|
654
|
+
that many and checks the latest, the file the recording checked. While
|
|
655
|
+
recording, a download check resolves `{env:...}`, `{data:...}` and
|
|
656
|
+
`{unique:...}` values, dates on the recording's clock, and values in a
|
|
657
|
+
pattern escaped, exactly as replay does.
|
|
658
|
+
A recording with a download needs this release; an older runner
|
|
659
|
+
refuses it by name.
|
|
660
|
+
|
|
8
661
|
## 1.0.0: Several people in one scenario
|
|
9
662
|
|
|
10
663
|
Released 2026-10-14.
|