saffron-ai 0.9.8 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +485 -0
- package/README.md +4 -4
- package/dist-pkg/cli.js +371 -240
- package/package.json +4 -3
- package/skills/saffron/SKILL.md +183 -87
- package/skills/saffron/references/config.md +10 -7
- package/skills/saffron/references/syntax.md +102 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,491 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.0.0: Several people in one scenario
|
|
9
|
+
|
|
10
|
+
Released 2026-10-14.
|
|
11
|
+
|
|
12
|
+
VS Code extension 0.7.0 and JetBrains plugin 0.7.0 (with saffron-ai
|
|
13
|
+
1.0.0): a CI run's row says what it left to view, and a results panel
|
|
14
|
+
shows a run as one page, with its screenshots and buttons for its traces,
|
|
15
|
+
scenarios, report and Import (VS Code: **Show Results**; JetBrains: below
|
|
16
|
+
the runs in the CI Runs tab).
|
|
17
|
+
|
|
18
|
+
Runner
|
|
19
|
+
|
|
20
|
+
- Capture names containing resolved environment secrets become stable opaque
|
|
21
|
+
identifiers in recordings and proposals, with every comparison updated to
|
|
22
|
+
match. Ambiguous names are refused instead of merging distinct captures.
|
|
23
|
+
- Live captured-value comparisons search child frames when no frame selector
|
|
24
|
+
was recorded, matching captures; explicit frame selectors stay scoped.
|
|
25
|
+
Comparisons select the first displayed non-empty value by locator order
|
|
26
|
+
and keep that source while polling, rather than passing on another
|
|
27
|
+
frame or fallback merely because its value differs. Removing a hidden
|
|
28
|
+
duplicate cannot shift the comparison onto another price. A removed or
|
|
29
|
+
replaced selected element fails the comparison rather than switching targets.
|
|
30
|
+
- Captured-value comparisons use displayed text, ignoring hidden descendants.
|
|
31
|
+
Changing only hidden text no longer passes a price-change assertion.
|
|
32
|
+
Captures and comparisons against a live element require a non-empty
|
|
33
|
+
displayed value; hidden or blank elements do not prove a value changed.
|
|
34
|
+
- **Browser sessions.** A step that starts `As "buyer"` runs in a
|
|
35
|
+
browser session of its own: a browser context with its own cookies,
|
|
36
|
+
storage, login and tabs. The session carries over to the steps after it
|
|
37
|
+
until another `As "..."` names one; steps before the first one run in
|
|
38
|
+
the default session (`As "default"` goes back to it), and the name may
|
|
39
|
+
come from an Examples column. A step that reads as a session step and
|
|
40
|
+
names none (`As "" ...`) stops the run with its file and line; `As
|
|
41
|
+
"Jane", I log in` stays an ordinary step. One
|
|
42
|
+
agent records, heals and refines across every session, its browser
|
|
43
|
+
tools following the session of the step it is on (every provider);
|
|
44
|
+
replay opens each session the first time a step needs it and goes back
|
|
45
|
+
to the page a session was on. Tab indexes and network checks count
|
|
46
|
+
within a session. Recordings store nothing new: a step's session
|
|
47
|
+
follows from the step texts, so recordings made before stay valid,
|
|
48
|
+
except for a step whose text already began `As "name" ` (with a space
|
|
49
|
+
after the quote): it now runs in that browser session, which starts
|
|
50
|
+
from `storageState` unless `sessions` gives it a login of its own.
|
|
51
|
+
Reword such a step, or configure the session.
|
|
52
|
+
- **A login per session.** `sessions` in `saffron.config.json`
|
|
53
|
+
(`{ "buyer": { "storageState": ".auth/buyer.json" } }`) starts a session
|
|
54
|
+
signed in as its own person; a session without an entry takes
|
|
55
|
+
`storageState`, as does the default session. Every file is checked
|
|
56
|
+
when the run starts, and so is the shape of each entry: a misspelt key
|
|
57
|
+
stops the run instead of starting that person as the default one, as
|
|
58
|
+
does a `sessions.default` naming a file other than `storageState` (the
|
|
59
|
+
default session never reads it). When the whole suite runs, a
|
|
60
|
+
configured session no step names gets a warning, and when two or more
|
|
61
|
+
sessions the steps name have no entry while `storageState` is set, the
|
|
62
|
+
run warns that they all start signed in as that one person.
|
|
63
|
+
- **A trace per session.** Each session is traced on its own
|
|
64
|
+
(`trace.buyer.zip` beside `trace.zip`, `proof-1.buyer.zip` beside
|
|
65
|
+
`proof-1.zip`), listed under its session's name in `saffron trace`, which
|
|
66
|
+
opens on the session a failure is in and marks the other sessions' steps
|
|
67
|
+
as theirs. The report tags the step where each session takes over and
|
|
68
|
+
names every trace file.
|
|
69
|
+
- **Editors.** Completion, hover, the similar-wording hint and the step
|
|
70
|
+
vocabulary (`saffron steps`, the MCP `search_steps`) offer a step without
|
|
71
|
+
its `As "..."`, so picking one never moves it into someone else's session.
|
|
72
|
+
Accepting a completion replaces the step's text, before and after the cursor, keeping
|
|
73
|
+
the keyword and any `As "..."`, and each suggestion's badge is the status
|
|
74
|
+
of the step as it will read there. A StepSet's preview keeps each step's
|
|
75
|
+
`As "..."`.
|
|
76
|
+
- Step reuse stays in a step's browser session, whether the step names
|
|
77
|
+
it or carries it over from an earlier `As "..."`: another person's
|
|
78
|
+
recording may type that person's own values (an address, a card), so it
|
|
79
|
+
never seeds a step for someone else, not even through a quoted value.
|
|
80
|
+
- A duplicate-wording rename, or a step edit the agent suggests, never
|
|
81
|
+
adds, drops or changes a step's `As "..."`; `saffron accept
|
|
82
|
+
--with-feature-edit` skips an edit that would, with the reason, and
|
|
83
|
+
`saffron diff` says so beside it.
|
|
84
|
+
- **The test folder is yours to name.** `saffron init` creates
|
|
85
|
+
the folder the `features` key names, and `init --examples` installs
|
|
86
|
+
the example suite into it, with its data in `dataDir`, instead of
|
|
87
|
+
`features/` and `data/`. `saffron status --json` names the folder as
|
|
88
|
+
`featuresDir`.
|
|
89
|
+
- **CI runs say what they left.** On GitHub, `saffron runs`
|
|
90
|
+
marks each run with its results and their size, a summary only,
|
|
91
|
+
results expired, none uploaded, or still running, from one listing of
|
|
92
|
+
the repository's artifacts (`results` in `--json`, with the words both
|
|
93
|
+
editors show as `results.label`). Azure Pipelines runs say so when
|
|
94
|
+
opened.
|
|
95
|
+
- **A summary of every run.** `saffron export --report-only` writes
|
|
96
|
+
`saffron-summary.zip`: the last run's report and screenshots, no
|
|
97
|
+
proposals and no traces, small enough to upload on every run. Viewing a
|
|
98
|
+
run (`report --run`) uses, for each shard and attempt, its full bundle
|
|
99
|
+
when there is one and its summary otherwise (`bundle.summaryOnly` when
|
|
100
|
+
only summaries were left); importing a run that left only summaries
|
|
101
|
+
says there is nothing to import. A full bundle, or a later attempt,
|
|
102
|
+
stands only for the same browser, base URL and shard: beside another
|
|
103
|
+
browser's or environment's results the view and import refuse the mix
|
|
104
|
+
by name instead of hiding a failure.
|
|
105
|
+
- **A CI run as one page.** Viewing a run writes its results page beside
|
|
106
|
+
its report (`panelHtml` in `report --run --json`; `--panel` prints it): totals, failed and pending-review scenarios as
|
|
107
|
+
cards with their screenshots, what passed, the proposals, and buttons
|
|
108
|
+
an editor wires to Open trace, Open scenario, Open report, Open run
|
|
109
|
+
page and Import.
|
|
110
|
+
- Agents other than Claude Code (Codex, Cursor, Antigravity) have their
|
|
111
|
+
tool calls run one at a time in the order they arrive, as
|
|
112
|
+
Claude Code runs them, so a step announced before a click is in effect
|
|
113
|
+
when the click runs; a call the vendor gave up on while it waited never
|
|
114
|
+
starts.
|
|
115
|
+
- **The Saffron plugin for Codex.** The plugin repo that serves Claude
|
|
116
|
+
Code is now a Codex marketplace too: `codex plugin marketplace add
|
|
117
|
+
s-chathuranga-j/saffron-claude-plugin`, then `codex plugin add
|
|
118
|
+
saffron@saffron`, installs the MCP server and the skill. Codex plugins
|
|
119
|
+
have no language server slot, so diagnostics and navigation stay
|
|
120
|
+
Claude Code only.
|
|
121
|
+
- **Node.js 22.12 or later.** The command-line library Saffron depends
|
|
122
|
+
on needs it, so an install that enforces engine versions failed on the
|
|
123
|
+
Node 20.6 the docs promised. Node 20 is past its end of life.
|
|
124
|
+
- **Playwright 1.64.** A recording session refuses `browser_tabs` with
|
|
125
|
+
`isolatedContext`: such a tab would run without the scenario's sign-in
|
|
126
|
+
and trace, and replay as a plain tab in the shared context.
|
|
127
|
+
|
|
128
|
+
Fixes:
|
|
129
|
+
|
|
130
|
+
- **An Outline row never passes on another row's data.** Recording and
|
|
131
|
+
the check that decides whether a shared recording fits a row now use
|
|
132
|
+
one rule for where a row value sits, so whatever the check would flag
|
|
133
|
+
is written as `<column>`: a value glued to other letters or digits
|
|
134
|
+
(`order<id>`, `2 <item>s booked`, `<name>님`), in any normalization, and
|
|
135
|
+
a value holding markup (`<script>...</script>`, `Alice <alice@x.com>`).
|
|
136
|
+
A value two columns share on the recorded row stays as typed (which
|
|
137
|
+
column it was cannot be told): a row typing them apart is recorded again
|
|
138
|
+
once, and where no row does (`ship` and `bill` both France, both Spain),
|
|
139
|
+
the recording is bound to its row. A value glued to other
|
|
140
|
+
text is written as `<column>` only in steps that use that column. Where one
|
|
141
|
+
recording cannot carry a row's value, it is bound to its row: a field
|
|
142
|
+
that still holds other rows' values beside the recorded row's
|
|
143
|
+
(`Red, Green, Blue`, `Plan changed: Basic to Pro`) stays as recorded,
|
|
144
|
+
and so does the row's value glued inside other text in a step that does
|
|
145
|
+
not name its column (`alice2024@example.com`). The recorded row, and
|
|
146
|
+
rows with the same value, replay; every other row fails red at zero
|
|
147
|
+
tokens, without healing, recording again or calling the agent, with a
|
|
148
|
+
message naming the column, the step and the fix: put `<column>` in the
|
|
149
|
+
step, give the text its own Examples column, or give that check its own
|
|
150
|
+
Scenario. A spelling recording cannot write binds the recording to its
|
|
151
|
+
row the same way, wherever the check that decides whether a recording
|
|
152
|
+
fits a row would find it: another case (`Welcome, Alice` for `alice`),
|
|
153
|
+
a slug (`/weather/new-york`), URL-encoded (`?q=C%2B%2B`), words joined
|
|
154
|
+
(`data-isopen` for `Is Open`). The other rows fail with guidance
|
|
155
|
+
instead of recording each other again forever. So does a value of one
|
|
156
|
+
or two characters that an assertion not naming its column checks
|
|
157
|
+
(`?page=1`, `Qty: 1`), which would otherwise pass on 10 or 12. A date
|
|
158
|
+
typed in ISO for a cell in another format makes the other rows record
|
|
159
|
+
again rather than pass, and so does a row filling columns the
|
|
160
|
+
recorded row left blank, and one whose shared value came from an
|
|
161
|
+
environment variable that has changed since. A recording that reads a
|
|
162
|
+
value through
|
|
163
|
+
`{data:...}` or `{env:...}` stays current when the data changes. The
|
|
164
|
+
recording format version no longer depends on the environment or clock.
|
|
165
|
+
The check also holds for a row added after recording, for `{data:...}`
|
|
166
|
+
and `{env:...}` cells, which count by the value they read, and for an
|
|
167
|
+
environment variable changed since recording, named in the cell or in
|
|
168
|
+
its `{data:...}` value (the recording keeps only a one-way hash of the
|
|
169
|
+
value, never the value). Recording and the check read a `{data:...}` value that is itself
|
|
170
|
+
`{env:...}` the way replay types it, data first, so such a row neither
|
|
171
|
+
records over the others forever nor passes on their values. A value the
|
|
172
|
+
step writes itself as the whole field, such as a login with `{env:...}`
|
|
173
|
+
or through its own `{data:...}` reference, never binds the row; a row
|
|
174
|
+
value glued to such a value (`{env:ADMIN}@example.com` typed as the
|
|
175
|
+
row's mail) still does. A login, a base URL (`{env:BASE_URL}/cart`) or text
|
|
176
|
+
the step spells in full counts as the step's own, so a row value that
|
|
177
|
+
happens to sit inside it never binds the row. A `{data:...}` reference
|
|
178
|
+
keyed by a column (`{data:users.<user>}`) is recorded as written, so each
|
|
179
|
+
row reads its own entry instead of the rows recording over each other;
|
|
180
|
+
when two such references choose the same entry on the recorded row, a
|
|
181
|
+
step that reads it without writing either one binds the recording to its
|
|
182
|
+
row, and editing that entry in the data file no longer makes the
|
|
183
|
+
recording stale. An expected response status or method, or an
|
|
184
|
+
attribute's name, follows its column in a step that names it (`the
|
|
185
|
+
response status is <status>`), where it was recorded as the first row's
|
|
186
|
+
and a row expecting another status could pass. A pending
|
|
187
|
+
proposal bound to its row refuses the other rows the same way, so they
|
|
188
|
+
never record over it, and the message names `saffron run --rerecord`
|
|
189
|
+
when the row it was recorded on is gone.
|
|
190
|
+
- Assertions read the first element a target matches. A first match that
|
|
191
|
+
is hidden (including `visibility:hidden`, a closed `<details>` or
|
|
192
|
+
`content-visibility:hidden`) gives way only to a shown one with the same
|
|
193
|
+
text, so "You have unsaved changes" can no longer pass a check for a
|
|
194
|
+
hidden "Saved". Expected text matches with whitespace collapsed first,
|
|
195
|
+
then as written; a blank expected text passes on a target that shows
|
|
196
|
+
none; a list check also matches the list as the page shows it
|
|
197
|
+
(`text-transform`). A column inside a frame selector is escaped on
|
|
198
|
+
replay.
|
|
199
|
+
- Linked folders inside the test folder are followed, each folder once,
|
|
200
|
+
so a link that loops no longer stops the run. A folder that cannot be
|
|
201
|
+
read, directly or through a link, stops it with the error, as in 0.9.9, instead of dropping its
|
|
202
|
+
scenarios and calling their recordings orphaned.
|
|
203
|
+
- An Outline row whose Examples cell is a `{data:...}`, `{unique:...}` or
|
|
204
|
+
`{date+N}` token, in any date format (`{date+2:DD.MM.YYYY}`), replays
|
|
205
|
+
its own value, not the first row's.
|
|
206
|
+
- A blank expected text or pattern (a `{data:...}` value or an Examples
|
|
207
|
+
cell that is now empty) passes only on an element that shows no text,
|
|
208
|
+
never on one showing any text.
|
|
209
|
+
- A response's expected status or method, or an attribute's name, follows
|
|
210
|
+
its `{data:...}` or `{env:...}` value: a data file changing 201 to 422
|
|
211
|
+
makes the recording read 422, and an older recording still holding 201
|
|
212
|
+
there is recorded again. A step reused for another quoted value (`"201"`
|
|
213
|
+
to `"422"`) takes the new value there too, or is not reused.
|
|
214
|
+
- An attribute check whose pattern takes only an empty value fails when
|
|
215
|
+
the attribute is missing: `disabled` checked as `^$` no longer passes
|
|
216
|
+
once the button is enabled, and neither does one taking any value (`.*`)
|
|
217
|
+
or anything but false (`^(?!false$)` for "is pressed"). A pattern that
|
|
218
|
+
takes an empty value and `false` but not `true`, such as `^(false)?$`
|
|
219
|
+
for "not invalid", still holds on a missing attribute, as in 0.9.9. A
|
|
220
|
+
recording that relied on another pattern holding on a missing attribute
|
|
221
|
+
now fails and needs recording again.
|
|
222
|
+
- `expectDiffers` without a target keeps the capture it compares as
|
|
223
|
+
recorded, like the capture that saves it, so other Outline rows no
|
|
224
|
+
longer fail with "no captured value named".
|
|
225
|
+
- A field that a blank cell or `{data:...}` value leaves empty no longer
|
|
226
|
+
makes a check hold for anything. An attribute's pattern left blank
|
|
227
|
+
passes only on an empty or missing attribute. An expected URL, a
|
|
228
|
+
response's URL, status, method or body, or the name or text of the
|
|
229
|
+
element a step finds, left blank for a row fails with a message naming
|
|
230
|
+
the field. A field that keeps other text around the blank value
|
|
231
|
+
(`Total: <amount>`, a pattern `\b<v>\b`) checks only that text.
|
|
232
|
+
- A list check (`Then the plan lists every {data:plans.<tier>}`) reads the
|
|
233
|
+
row's own list, and one recorded before, still reading the first row's,
|
|
234
|
+
is recorded again.
|
|
235
|
+
- A column a step names follows its row inside an attribute's name too
|
|
236
|
+
(`data-<tier>`), in any case (`disabled` for a row's `Disabled`), so two
|
|
237
|
+
rows no longer record each other again forever. A response's status or
|
|
238
|
+
method, or an attribute's name, takes a `{data:...}` reference only when
|
|
239
|
+
it is that whole value and the step names the reference: a 200 is not
|
|
240
|
+
an order's quantity of 200. A captured value's name (`captureText` and
|
|
241
|
+
`expectDiffers`) no longer counts as a row's value, which also had two
|
|
242
|
+
rows recording each other again forever.
|
|
243
|
+
- A `{data:...}` reference that two references chose alike on the
|
|
244
|
+
recorded row binds the recording to its row when a response status,
|
|
245
|
+
method or attribute holds it too, as it does elsewhere.
|
|
246
|
+
- A step reused for another quoted `{data:...}` entry, or for an edited
|
|
247
|
+
doc string, is not reused while the old entry's value or the old doc
|
|
248
|
+
string is still written in other text (a test id `row-alice-smith`) or
|
|
249
|
+
the old reference is. An edited doc string still written that way makes
|
|
250
|
+
the recording stale too.
|
|
251
|
+
- An Outline recording made by 0.9.x, which does not name its row, is
|
|
252
|
+
recorded again once on a row it holds another row's value for. Under
|
|
253
|
+
`--no-agent`, as in CI, that row stays red, with a message saying the
|
|
254
|
+
recording comes from an older saffron-ai, until it is recorded once.
|
|
255
|
+
- A response status, method or attribute that a step not naming the
|
|
256
|
+
column checks as the row's value (`Then the API answers as expected`,
|
|
257
|
+
recorded as 201 on the row whose `<code>` is 201) binds the recording to
|
|
258
|
+
its row for every column holding that value, so the other rows fail with
|
|
259
|
+
guidance instead of checking 201. So does one holding what a `{data:...}`
|
|
260
|
+
reference keyed by columns reads on that row
|
|
261
|
+
(`{data:users.<region>.<user>}` reading 201), for the columns it reads.
|
|
262
|
+
Both are judged on the recorded row alone, so a row added after the
|
|
263
|
+
recording was accepted is refused too. So does an attribute's name or a
|
|
264
|
+
list check's data path holding the row's value, in any case or as a
|
|
265
|
+
slug: an attribute's name as a whole part (`aria-expanded` for a row's
|
|
266
|
+
`Expanded`, `data-is-open` for `Is Open`, never `aria-invalid` for
|
|
267
|
+
`Ali`), a data path inside a key too (`{data:plans.basicPlan}` for
|
|
268
|
+
`basic`). Where the step names the column, the value follows the row
|
|
269
|
+
instead when it is a whole part of the name or path: an attribute's
|
|
270
|
+
name in any case (`data-<tier>`), a data path spelled as the cell is
|
|
271
|
+
(`{data:plans.<tier>}`). Inside a word it is never written as
|
|
272
|
+
`<column>` (`aria-invalid` stays as it is on a row whose value is
|
|
273
|
+
`valid`). A value that only coincides (an
|
|
274
|
+
amount of 200 beside a check of status 200, a `post` beside method POST,
|
|
275
|
+
a `disabled` state beside the `disabled` attribute) binds too: give the
|
|
276
|
+
check its own Scenario, or give the expected value its own column.
|
|
277
|
+
- The text or regex inside a fallback or frame selector that is blank for
|
|
278
|
+
a row (`button:has-text("<v>")`, an XPath `contains(., '<v>')`,
|
|
279
|
+
`text=<v>`, anywhere in a `>>` chain) fails with a message naming the
|
|
280
|
+
selector, instead of matching any element. An exact attribute match
|
|
281
|
+
such as `input[value="<v>"]`, with or without the `i` or `s` flag, still
|
|
282
|
+
checks for an empty value.
|
|
283
|
+
- A failed `Then` stays red when a pending proposal from `--rerecord`
|
|
284
|
+
checks something else there; it no longer turns yellow. The same holds
|
|
285
|
+
when the run fails before reaching the `Then`: a pending proposal covers
|
|
286
|
+
it only if it checks every `Then` the assertion policy protects as the
|
|
287
|
+
committed recording does. A heal never files an edit to the text of a
|
|
288
|
+
`Then` step.
|
|
289
|
+
- An `{env:...}` value with a `'` or regex characters in it (an email) is
|
|
290
|
+
masked in a failed step's message, reports, proposals and traces, where
|
|
291
|
+
Playwright writes it escaped.
|
|
292
|
+
- Data-table and doc-string parameterization preserves captured-value names
|
|
293
|
+
in comparisons. Editing a table value no longer makes replay look for a
|
|
294
|
+
different capture that was never saved.
|
|
295
|
+
- Network body assertions wait for a successfully read, complete response.
|
|
296
|
+
A pending or failed body read no longer counts as an empty body, and
|
|
297
|
+
responses longer than 262,144 characters are checked in full instead of
|
|
298
|
+
letting a truncated prefix pass the assertion. A body check against a
|
|
299
|
+
response that is not JSON, text or XML (an image, a binary file) fails
|
|
300
|
+
instead of matching an empty body.
|
|
301
|
+
- **Recording format version 2.** A recording with several people
|
|
302
|
+
(`As "buyer"` steps), or one that uses a double, right or modified
|
|
303
|
+
click, a multi-select or a cleared selection (all now recorded and
|
|
304
|
+
replayed as made), is written as version 2, so an older runner refuses
|
|
305
|
+
it instead of replaying everyone in one browser or a plain click. So is
|
|
306
|
+
an Outline recording bound to its row or keeping a recorded row's value
|
|
307
|
+
as typed, which only a runner that checks the Examples row replays
|
|
308
|
+
safely, and one whose response status, method or attribute name follows
|
|
309
|
+
a column or holds a `{data:...}`, `{env:...}`, `{date...}` or
|
|
310
|
+
`{unique:...}` token, which a 0.9.x runner would read literally. Every
|
|
311
|
+
other recording stays version 1.
|
|
312
|
+
- Recordings 0.9.9 saved for names with no ASCII letters (`unnamed.json`)
|
|
313
|
+
are still found, a damaged older recording is reported as invalid
|
|
314
|
+
instead of recorded again, and `saffron prune` keeps an older evidence
|
|
315
|
+
folder its live scenario still owns until the scenario has run under
|
|
316
|
+
the new name. `saffron runs` works with a `gh` too old to report when a
|
|
317
|
+
re-run started.
|
|
318
|
+
- `saffron steps`, `status` and the MCP `search_steps` count a step
|
|
319
|
+
recorded in any person's session as recorded, and divergence is counted
|
|
320
|
+
per session; `search_steps` never shows a session's `As "..."` text.
|
|
321
|
+
The MCP tools read `saffron.config.json` on every call, so a changed
|
|
322
|
+
test folder is seen without restarting the server.
|
|
323
|
+
- A dropdown option whose text holds `]` (`Large [L]`) is recorded whole.
|
|
324
|
+
A `{date+N}` Examples cell is matched only as a whole value, only in
|
|
325
|
+
the steps that use it, in any format including two-character ones
|
|
326
|
+
(`{date+1:DD}`) and inside regex and selector fields.
|
|
327
|
+
- `expectEach` reads the same shown text as the other text checks: a
|
|
328
|
+
visible value is found even when a hidden option has the same text, and
|
|
329
|
+
a hidden line break no longer splits a word in two. Only a line the
|
|
330
|
+
page draws ends an entry (a line break in the HTML source does not), and
|
|
331
|
+
text inside a closed `<details>`, `hidden="until-found"` or
|
|
332
|
+
`content-visibility: hidden` is not shown. A check aimed at a dropdown
|
|
333
|
+
option, a `<textarea>`'s text, or a visible part of a hidden container
|
|
334
|
+
passes again, and text checks are about ten times faster on deep pages.
|
|
335
|
+
A text or visibility check reads the first copy of its target the page
|
|
336
|
+
shows, skipping a hidden duplicate (a hidden mobile menu before the
|
|
337
|
+
desktop one), and never another copy further down. A two-line doc
|
|
338
|
+
string or a value with doubled spaces matches the page's text as it is
|
|
339
|
+
shown, and text below the fold in a `content-visibility: auto` box is
|
|
340
|
+
found in WebKit too.
|
|
341
|
+
- A `{date+N}` Examples cell passed through a step's table or doc string
|
|
342
|
+
replays each row's own date. When two Examples columns (or two keys of a
|
|
343
|
+
table) hold the same value in the recorded row, that value stays as
|
|
344
|
+
typed rather than being guessed into one of them: a later row with
|
|
345
|
+
different values records again instead of passing on the wrong column.
|
|
346
|
+
A recording keeps the Examples row it was made on (`examples` in the
|
|
347
|
+
recording), so a row whose value differs from one left as typed is
|
|
348
|
+
recorded again rather than replayed. A date spelled with its year
|
|
349
|
+
follows its row in later steps too; dates resolve on the clock the
|
|
350
|
+
recording started on. Step reuse applies the same check, so a step
|
|
351
|
+
still holding another row's value is never reused for a row that
|
|
352
|
+
differs, and a one- or two-character value (`1`, `2`) that fills a
|
|
353
|
+
whole field follows its row too, including as the whole of a regex
|
|
354
|
+
(`^12$`). An Examples cell left blank on purpose replays blank, in the
|
|
355
|
+
fill and in the assertion that checks it, instead of being reported as
|
|
356
|
+
a lost value.
|
|
357
|
+
- An Outline assertion keeps its whole recorded text with only the row's
|
|
358
|
+
value made a placeholder (`Order for <user>: cancelled`, not just
|
|
359
|
+
`<user>`), so a row cannot pass on a check cut down to its value. A
|
|
360
|
+
value the step writes out itself (`I log in as "alice"`) is never bound
|
|
361
|
+
to an Examples column, a value inside a CSS selector (`#row-alice`)
|
|
362
|
+
follows its row, and step reuse never carries one column's placeholder
|
|
363
|
+
into another scenario, nor reuses a step whose recording still names
|
|
364
|
+
the old value in another spelling (`Red Shoe` as `/product/red-shoe`).
|
|
365
|
+
Recordings made by 0.9.9 are not rewritten; a newly recorded Outline
|
|
366
|
+
whose rows differ in such text records each row's check as it is.
|
|
367
|
+
- A CSV line of only commas or spaces is no row (write `"",""` for a
|
|
368
|
+
deliberate empty row). An empty `"features"` means `features/`
|
|
369
|
+
everywhere.
|
|
370
|
+
- A re-run GitHub run's results label describes what its latest attempt
|
|
371
|
+
left, shard by shard. In a sharded workflow, name each summary
|
|
372
|
+
`saffron-summary-<shard>` (the docs' snippet now says so).
|
|
373
|
+
- Text checks pass only on text the page shows: hidden text, a hidden part
|
|
374
|
+
of a visible element or a hidden dropdown option no longer satisfies
|
|
375
|
+
"should be shown", and text in a `display: contents` wrapper does. A
|
|
376
|
+
check against a `{data:...}` list (`expectEach`) leaves out hidden
|
|
377
|
+
options and hidden option groups too.
|
|
378
|
+
- A filled value that looks like code (`text .click('x')`) is recorded as
|
|
379
|
+
the fill it is; two confirm dialogs in one step each keep their own
|
|
380
|
+
handler.
|
|
381
|
+
- Step reuse refuses a quoted-value match when the old value survives in
|
|
382
|
+
a test id, selector or frame (`I delete "ann"` never clicks
|
|
383
|
+
`delete-bob`), and when the recording's locators name a placeholder
|
|
384
|
+
the scenario lacks.
|
|
385
|
+
- `saffron report --merge` checks every shard history line field by
|
|
386
|
+
field, and the dashboard escapes every value it shows.
|
|
387
|
+
- A proposal's feature edit that would leave a step empty is refused.
|
|
388
|
+
- Healing a mid-scenario assertion (`adaptable-mid`) keeps what the step
|
|
389
|
+
already did, such as a remembered value.
|
|
390
|
+
- Secrets stay masked in a failed proof's tab and response URLs (report,
|
|
391
|
+
bundle) and in both terminal lines a pending proposal prints when it no
|
|
392
|
+
longer replays.
|
|
393
|
+
- `saffron author` with Claude runs with no built-in tools.
|
|
394
|
+
- A recording without `recordedAt` no longer stops step reuse.
|
|
395
|
+
- A date the scenario writes out (`I enter 2026-10-10`, a table, a data
|
|
396
|
+
value, an `{env:...}` variable) is entered as written; only a date the
|
|
397
|
+
agent picked moves with the calendar.
|
|
398
|
+
- Typed text keeps its line breaks and tabs; double, right and modified
|
|
399
|
+
clicks replay as recorded; a multi-select keeps every option.
|
|
400
|
+
- A browser-matrix bundle's evidence (`.../chromium/failure.png`) is
|
|
401
|
+
accepted on view and import.
|
|
402
|
+
- `features: "."` finds only real files, never `.saffron/`, `node_modules`
|
|
403
|
+
or `.git`, and `admin/login.feature` and `shop/login.feature` get a
|
|
404
|
+
recording folder each instead of sharing one (so does a file outside
|
|
405
|
+
the `features` folder). Recordings saved in the shared folder are
|
|
406
|
+
still found by the feature they name; files directly in the folder
|
|
407
|
+
keep their paths.
|
|
408
|
+
- A browser session named by an Examples column (`As "<who>"`) never
|
|
409
|
+
seeds another row's step: the recording made for the buyer's row is
|
|
410
|
+
not reused for the seller's.
|
|
411
|
+
- Scenario names with letters outside ASCII (`Login Успешный`, `Café
|
|
412
|
+
ouvert`) get distinct recording files and screenshot folders;
|
|
413
|
+
recordings saved under the earlier name are still found.
|
|
414
|
+
- Completing a step in the middle of a line, or before its `As "..."`,
|
|
415
|
+
never duplicates or drops text.
|
|
416
|
+
- `saffron status --json` says when the orphan scan could not finish
|
|
417
|
+
(`orphanScan: { complete, errors }`); both editors show it instead of
|
|
418
|
+
"no orphaned recordings".
|
|
419
|
+
|
|
420
|
+
Agent skill and Claude Code plugin
|
|
421
|
+
|
|
422
|
+
- The skill now teaches a loop for writing a new test: survey the
|
|
423
|
+
project, search the existing steps, place the scenario (StepSet or
|
|
424
|
+
inline), fix the language server's diagnostics, run only that
|
|
425
|
+
scenario, review and accept. It says to write steps as a user sees the
|
|
426
|
+
page, never as selectors or test ids, and the syntax reference ends in
|
|
427
|
+
one complete example project.
|
|
428
|
+
- The plugin's listing describes what Saffron does and its README states
|
|
429
|
+
the license terms in plain words.
|
|
430
|
+
|
|
431
|
+
## 0.9.9: A browser matrix in one run
|
|
432
|
+
|
|
433
|
+
Released 2026-10-06.
|
|
434
|
+
|
|
435
|
+
Runner
|
|
436
|
+
|
|
437
|
+
- **A browser matrix in one run.** `saffron run --browser
|
|
438
|
+
chromium,firefox,webkit` (or a list in `browser`) runs the suite in each
|
|
439
|
+
browser, one after another, into one report: a row per scenario per
|
|
440
|
+
browser, in the terminal, the report, the history and the dashboard.
|
|
441
|
+
Chromium-family browsers go first, so Firefox and WebKit replay what
|
|
442
|
+
they recorded. Every browser starts before anything runs. Screenshots
|
|
443
|
+
and traces are kept per browser, and `saffron trace` names them
|
|
444
|
+
`feature:scenario@browser`. The dashboard stands a scenario at its
|
|
445
|
+
worst browser and never calls a difference between browsers flaky.
|
|
446
|
+
Refused with `--shard`. Every result now names its browser, so running
|
|
447
|
+
one browser again replaces that browser's rows and keeps the others'.
|
|
448
|
+
`saffron status --json` (and the MCP `project_status` tool) names the
|
|
449
|
+
browser of each row that needs attention.
|
|
450
|
+
- **Device and viewport profiles.** `--device "iPhone 15"` (config
|
|
451
|
+
`device`) runs every scenario under a Playwright device profile,
|
|
452
|
+
recordings and heals included; `--viewport 1280x720` (config `viewport`)
|
|
453
|
+
sets the size alone. An unknown device names the closest ones, and a
|
|
454
|
+
mobile profile in Firefox stops the run up front.
|
|
455
|
+
- **Parallel recording and healing.** `--agent-workers N` (config
|
|
456
|
+
`agentWorkers`, default 1) lets up to N agent sessions run at once,
|
|
457
|
+
each in a browser of its own. An outline's rows stay together on one
|
|
458
|
+
worker. Each session is paid for: raise it deliberately.
|
|
459
|
+
- **Shards know their browser.** A shard records the browser it ran in;
|
|
460
|
+
`saffron report --merge` refuses shards of different browsers, a
|
|
461
|
+
re-run attempt replaces only its own browser's shard, and each
|
|
462
|
+
browser's merge keeps its own history line.
|
|
463
|
+
- **A bad storage state stops the run first.** A `storageState` file that
|
|
464
|
+
is missing, unreadable or not JSON now stops the run at startup, exit 2,
|
|
465
|
+
before `--rerecord` deletes any recording.
|
|
466
|
+
- **No second replay without an agent.** With `--workers` and
|
|
467
|
+
`--no-agent`, or in Firefox and WebKit, a scenario that went red in the
|
|
468
|
+
parallel phase is no longer run again afterwards: nothing could change
|
|
469
|
+
its result.
|
|
470
|
+
- **Per-worker timing.** The report has a timing panel (each worker's
|
|
471
|
+
runs, busy time and share, and the five slowest scenarios), and the
|
|
472
|
+
terminal prints each worker's share and the three slowest scenarios.
|
|
473
|
+
Merged shards list each machine's workers.
|
|
474
|
+
- **Saffron's own data in the replay.** Each step shows the tabs it
|
|
475
|
+
opened or switched to and the response each network check matched,
|
|
476
|
+
and the Saffron pane shows what the agent cost for the scenario. The
|
|
477
|
+
report's steps carry `tabs` and `responses` for tools.
|
|
478
|
+
- **Browsers in the editors' contracts.** `saffron status --json` and
|
|
479
|
+
`saffron report --run <id> --json` give each row its `browser`, and the
|
|
480
|
+
run's `browsers` when its rows span several.
|
|
481
|
+
|
|
482
|
+
VS Code extension 0.6.2
|
|
483
|
+
|
|
484
|
+
- After a browser-matrix run, Last Run and a CI run's results label each
|
|
485
|
+
row with its browser, and Open Replay and Open Trace open that
|
|
486
|
+
browser's execution alone (`feature:scenario@browser`).
|
|
487
|
+
|
|
488
|
+
JetBrains plugin 0.6.2
|
|
489
|
+
|
|
490
|
+
- The same in the Last Run and CI Runs tabs: rows say their browser, and
|
|
491
|
+
the replay opens that browser's execution alone.
|
|
492
|
+
|
|
8
493
|
## 0.9.8: Run one scenario, from the editor too
|
|
9
494
|
|
|
10
495
|
Released 2026-10-06.
|
package/README.md
CHANGED
|
@@ -44,7 +44,7 @@ of short `.saffron` scenarios against the public demo shop at saucedemo.com
|
|
|
44
44
|
the middle of scenarios) so the first run records and replays something
|
|
45
45
|
real.
|
|
46
46
|
|
|
47
|
-
**Built for the AI era:** Claude Code users install everything at once from the [Saffron plugin](https://github.com/s-chathuranga-j/saffron-claude-plugin) (`/plugin marketplace add s-chathuranga-j/saffron-claude-plugin`, then `/plugin install saffron@saffron`): the language server for diagnostics and navigation while Claude edits `.saffron` files, the MCP server, and the skill. For every other agent, the package ships an [Agent Skill](https://agentskills.io) that teaches Claude Code, Codex, Cursor, Copilot and any skill-aware agent to write Saffron tests efficiently. Reuse recorded wordings, keep assertions sacred, secrets as `{env:VAR}`, so the tests your agents write replay at zero tokens instead of paying for near-duplicate recordings.
|
|
47
|
+
**Built for the AI era:** Claude Code users install everything at once from the [Saffron plugin](https://github.com/s-chathuranga-j/saffron-claude-plugin) (`/plugin marketplace add s-chathuranga-j/saffron-claude-plugin`, then `/plugin install saffron@saffron`): the language server for diagnostics and navigation while Claude edits `.saffron` files, the MCP server, and the skill. Codex users get the MCP server and the skill from the same repo (`codex plugin marketplace add s-chathuranga-j/saffron-claude-plugin`, then `codex plugin add saffron@saffron`). For every other agent, the package ships an [Agent Skill](https://agentskills.io) that teaches Claude Code, Codex, Cursor, Copilot and any skill-aware agent to write Saffron tests efficiently. Reuse recorded wordings, keep assertions sacred, secrets as `{env:VAR}`, so the tests your agents write replay at zero tokens instead of paying for near-duplicate recordings.
|
|
48
48
|
|
|
49
49
|
CI tip: `saffron run --strict` treats yellow (passed-with-adaptation) as a
|
|
50
50
|
failure until its cache proposal is reviewed: cached-green-only builds.
|
|
@@ -73,7 +73,7 @@ node dist/cli/index.js -p examples accept --all
|
|
|
73
73
|
|
|
74
74
|
| Command | What it does |
|
|
75
75
|
|---|---|
|
|
76
|
-
| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
|
|
76
|
+
| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b[,b...]] [--device name] [--viewport WxH] [--no-hooks] [--workers n] [--agent-workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
|
|
77
77
|
| `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
|
|
78
78
|
| `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
|
|
79
79
|
| `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
|
|
@@ -155,7 +155,7 @@ milestones, statuses and decision gates, and everything shipped so far in
|
|
|
155
155
|
its "Where we are" table.
|
|
156
156
|
Headlines ahead: API testing (reusable REST operations, fixtures and schema
|
|
157
157
|
validation), scenarios with two users in two browsers, a configurable
|
|
158
|
-
features folder,
|
|
158
|
+
features folder, failure video,
|
|
159
159
|
agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
|
|
160
160
|
(gated on Level-1 divergence data), and the long-term open-core SaaS layer.
|
|
161
161
|
|
|
@@ -168,4 +168,4 @@ agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
|
|
|
168
168
|
- **Docs site**: `npm run docs` builds a self-contained page at
|
|
169
169
|
## License
|
|
170
170
|
|
|
171
|
-
Free to use, including commercially, under the Saffron Free Use License (
|
|
171
|
+
Free to use, including commercially, under the [Saffron Free Use License](https://saffron-ai.io/docs/license) (the [LICENSE](LICENSE) file in this package). The runner's source is closed.
|