saffron-ai 1.3.3 → 1.3.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,324 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
5
5
  JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
6
6
  release heading names its main feature: `## <version>: <title>`.
7
7
 
8
+ ## 1.3.5: Scenarios kept from running at once
9
+
10
+ Released 2026-11-08.
11
+
12
+ Runner
13
+
14
+ - `@serial:<name>` keeps scenarios that would trip over each other's data
15
+ from running at the same time. With `--workers` or `--agent-workers`,
16
+ scenarios that share a name run one at a time, in file order, while the
17
+ rest of the suite runs beside them; a plain `@serial` is one shared name.
18
+ On a Feature line it covers the whole file. `--shard` keeps every
19
+ scenario under one name on the same machine, so two machines never run
20
+ them at once either.
21
+ - A value is no longer found inside a number grouped by spaces or
22
+ apostrophes: `250 kr` passed on `1 250 kr`, and `1 250` on
23
+ `1 250 000`. Nor is an amount found inside another currency written with
24
+ the same symbol: `$5.00` passed on `R$5.00` and `CA$5.00`. Both were
25
+ false greens, in text, pattern and download checks.
26
+ - A response check's row value is no longer found inside a longer word
27
+ written percent-encoded in the URL: `Jos` passed on `/users/Jos%C3%A9`.
28
+ - A captured decimal keeps the digits the API sent: `0.123456789012345678`
29
+ was rounded, so a `differs from {captured:...}` check passed on an
30
+ unchanged value, and a later request sent another number.
31
+ - `should have a non-empty "/id"` holds on a number: a numeric id was red
32
+ on every run.
33
+ - The recording agent no longer sees a base URL's password: the APIs'
34
+ summary in its prompt, a refused request and a request's errors quoted
35
+ it. A secret percent-encoded the way Python writes it (`%28`, `%2A`) is
36
+ masked too.
37
+ - The dashboard, `saffron status` and the CI run panel count reds the app
38
+ did not cause (a stale recording in a run with `--no-agent`) as *not
39
+ run*, as the HTML report does. The dashboard counted one as failed, and
40
+ said it had been red since before its grid. A scenario whose latest
41
+ result is such a red now counts as not run, never as passing on an older
42
+ green: the dashboard said 100% passing after a run that exited 1.
43
+ - "Not shown" no longer passes while the text is on screen beside a
44
+ number: `iPhone 15` was taken for part of `15 128` in
45
+ `iPhone 15 128 GB`. A positive check reads a number as grouped with the
46
+ next only after one to three digits that are not part of a word, so
47
+ `Galaxy S24` is found in `Galaxy S24 256 GB` again.
48
+ - Thai, Lao, Khmer and Myanmar digits keep a number whole (`1` passed on
49
+ `12` written in Myanmar digits), as do the Arabic decimal point and
50
+ thousands separator (`١٫٥`) and a thin space between thousands in a
51
+ downloaded file.
52
+ - A base URL password holding a raw `@` is masked whole: its tail after
53
+ the `@` reached the recording agent.
54
+ - An untyped `differs from` holds a number however the step writes it: an
55
+ unchanged `10.5` passed against `10.50`, and `100` against `1e2`.
56
+ - A captured list or object keeps each number as the API wrote it. A
57
+ decimal JavaScript cannot hold was sent on as a string; it is now
58
+ refused, as the same number captured alone is.
59
+ - `@serial` holds while a heal proves an Outline's other rows: a row with
60
+ its own `@serial` Examples tag was replayed without its name. Tags are
61
+ read in any case, with commas between names (`@Serial:Rooms` and
62
+ `@serial:a,b` were ignored), and `saffron eject` writes `workers: 1`
63
+ for a suite with `@serial` scenarios, or warns when the project has its
64
+ own Playwright config.
65
+ - "Not shown" counts a number beside a comma as shown: `2` passed while
66
+ `Tags: 1,2,3` was on screen, and `42` beside `1001,42`. It also counts
67
+ `$5.00` as shown in `US$5.00`, and a role name pattern with a row value
68
+ reads `15 128` as one number, as a text target does.
69
+ - `256 GB` is found in `Galaxy S24 256 GB`: a model number heads no
70
+ grouped number.
71
+ - The dashboard counts a scenario whose latest run was not run against
72
+ the pass rate: one scenario passing while the rest were not run read
73
+ 100%, 1 of 1.
74
+ - A `differs from` checked as a string compares a number too large for
75
+ JavaScript by its digits: the same int64 id passed.
76
+ - A base URL password holding a quote or a space is masked before the
77
+ recording agent reads the base URL.
78
+ - The exported helper refuses `$10.50` for a role name recorded as
79
+ `Refund of $10.`, keeps a captured object's numbers as the API wrote
80
+ them, and `saffron eject` gives the trace warning beside the
81
+ `@serial` one, where it gave only the `@serial` one.
82
+ - An Outline's "not shown" check on a row value the recording glued into
83
+ a word reads the word in any form. Recorded on `apples` as `<item>s`,
84
+ row `box` checked that `boxs` was not shown and passed while `boxes`
85
+ was listed; it now counts `boxes` as shown.
86
+ - An exported "not shown" check looks in the app's own frames, as replay
87
+ does: it passed while `Card declined` showed in a same-origin iframe. It
88
+ also waits for the page's requests and for the hidden state to hold, as
89
+ replay does: it passed before an error drawn 300 ms after a click.
90
+ - The dashboard keeps every browser of the last CI run's matrix, where
91
+ each browser's shards are merged on their own: the browser merged last
92
+ hid the others' reds. A red from before browsers were recorded stands
93
+ when a later run is not run in one browser and passes in another.
94
+ - A `differs from` checked as a string fails on the same number, `true`
95
+ or `null`: it passed on any of them.
96
+ - `saffron eject` run again no longer warns about `@serial` when the
97
+ config it wrote already runs one test at a time.
98
+ - `run_plan` (MCP) and `saffron run --plan` name each `@serial` name and
99
+ how many tests share it, and the agent skill's syntax reference lists the
100
+ tags that change how a scenario runs.
101
+
102
+ ## 1.3.4: Checks that wait for a settled page, and credentials kept masked
103
+
104
+ Released 2026-11-07.
105
+
106
+ Runner
107
+
108
+ - An Outline row no longer passes on another row's data in three more
109
+ ways: a check naming a column that no step of the recording writes ("the
110
+ `<plan>` plan shows ...") binds the recording to its row, a row's number
111
+ is found where the page shows it grouped or with a decimal comma ("Up
112
+ to $1,000" for 1000), and a request body's number is taken for a
113
+ constant only where the body itself writes the column. A style check's
114
+ property is a value a row can hold. A check run in the session a column
115
+ names (`As "<who>"`) follows that column; one run in another session
116
+ (`As "admin"`) does not, and a step writing the column counts only when
117
+ it comes before the check in the same session. A row's number is also
118
+ found negative ("−$1,000") and grouped by an apostrophe or a no-break
119
+ space ("CHF 1'000").
120
+ - An amount check fails when the target shows another currency than the
121
+ captured amounts, and captures in two currencies are refused. A symbol
122
+ is compared with the codes it can stand for: "$5" may be USD 5 or CAD 5,
123
+ never NOK 5 or EUR 5, "€" is EUR, and a dollar its letters name is that
124
+ dollar ("US$", "CA$", "A$", "R$" and others); every capture is compared
125
+ with every other. A lowercase word that is also a code ("Top 10",
126
+ "150 eur") is compared with codes only. A URL check reads a
127
+ path whole past `-`, `.`, `_`, `~` and `+`, and past `:` before a letter
128
+ ("/checkout" is not "/checkout-failed" or "/checkout_failed"; a host
129
+ before its port, "localhost:3000", still matches), and a word starts as
130
+ it ends: after a letter and a separator it is part of another ("paid"
131
+ is not in "/orders/not-paid"; in a query "+" and a space still separate
132
+ words, so "shoes" is in "?q=red+shoes"), a number also after a
133
+ digit ("42" is not in "/1-42"), while an id-prefixed slug keeps its words
134
+ ("blue-shirt" is in "/products/42-blue-shirt") and "example.com" is in
135
+ "shop.example.com". A doubled separator ("/checkout--failed") is one, in
136
+ replay and in specs `saffron eject` writes. A committed check of a path
137
+ the app shows with a slug after it ("/products/42" on
138
+ "/products/42-blue-shirt"), or of one word inside a slug ("shirt" or
139
+ "US" in "/en-US/cart"), fails after upgrading: record it again with the
140
+ whole slug, or check the id another way. A currency code is read beside the
141
+ number, capitals first, so "Top up NOK 150" is in NOK, and "NET 150
142
+ EUR", "150 eur" and "EUR (150.00)" are in euros. Text inside `<noscript>`, or inside an SVG
143
+ definition never drawn where it stands, is not shown text, whether the
144
+ check's target is the container or that element itself.
145
+ - API tests: a group's `same` check compares values exactly (two int64
146
+ payment references a digit apart differ, as do 0.1 and
147
+ 0.10000000000000001, and a number and a string of its digits), a number
148
+ in a body that JSON cannot hold as written is refused, in any spelling
149
+ (`9007199254740993`, `9007199254740993.0`, `9.007199254740993e15`,
150
+ `0.10000000000000001`; it would be sent rounded to another value: send
151
+ it as a string). A row or capture that held such a number and passed
152
+ before is refused after upgrading, a `differs` check whose row value is
153
+ empty is refused, an expected decimal with more digits than a number
154
+ holds is refused, and a group's contract check ties the recording to
155
+ the contract, so `api pin` re-records it.
156
+ - A provider's daily quota, "exhausted your capacity" and "hit your
157
+ limit" stop the run's AI sessions as other usage limits do. A key a
158
+ vendor quotes with its middle starred or elided (`sk-...XwVu`) is
159
+ masked, every Cursor error masks CURSOR_API_KEY, including one the SDK
160
+ throws, and an Antigravity error that ends in JSON names its
161
+ message. A `verify` scenario path that leaves the project through a
162
+ link is refused.
163
+ - Specs `saffron eject` writes open web pages only (never `data:`,
164
+ `file:` or `javascript:`) and upload only the project's own files (no
165
+ dotfiles, nothing outside the project, no link to a dotfile or out of
166
+ the project, no hard link, no folders), as replay does; they refuse a
167
+ `differs` whose value a blank row or capture empties, and a body number
168
+ they cannot send exactly. An Examples cell holding such a
169
+ number marks its test fixme instead of stopping the eject.
170
+ - A `"""` or a triple backtick at the start of a description line no
171
+ longer opens a doc string, which left every `StepSet` line after it
172
+ unread: as in Gherkin, a doc string opens only after a step in a
173
+ scenario, Background, StepSet or Fixture, never in a Feature, Rule or
174
+ Examples description, in the runner and the language server alike.
175
+ - An API that answers with a refused connection in its body or a field
176
+ (its own database's) is a failure, never "not run, environment
177
+ missing": only a request that got no response, or the browser's own
178
+ navigation error, says nothing answered.
179
+ The dashboard's recent failures pass over reds the app did not cause
180
+ when telling whether a scenario still fails and for how long.
181
+ - Recording: a credential an API handed out during the session is masked
182
+ in everything the browser tools show the agent (a token in a page URL
183
+ was shown whole, and an earlier value of a credential that changed); a
184
+ page the agent opens with an `{env:...}` value in its URL must be on the
185
+ app's origin or one the scenario writes, a URL with no scheme read as
186
+ the browser opens it; code a check runs through `browser_evaluate` that
187
+ clicks, submits, sets a value, navigates or sends a writing request is
188
+ refused, as a click in a check is;
189
+ with a browser already running (`transport.browser` or a
190
+ `transport.module`), recordings run one at a time, as two sessions on
191
+ one browser closed each other's pages.
192
+ - A password or user name in a base URL is masked wherever a URL quotes
193
+ it (`https://qa:***@staging.test`): the report's errors, the run
194
+ history, proposals, `status`, `diff`, bundles, traces and the API
195
+ transcript, and the agent is never shown it. Only there: the same word
196
+ elsewhere in a message is left as it is.
197
+ - A local notation beside the number (`100 kr`, `Rs. 500`, `5,00 zł`) is a
198
+ currency of its own in an amount check, so captures in "kr" fail a total
199
+ shown in euros. A URL check reads a percent-encoded letter beyond a
200
+ separator as a letter ("/checkout" is not in "/checkout-%C3%A9chou%C3%A9").
201
+ - A proposal or recording the run cannot read (no permission, a folder in
202
+ its place) is listed as unreadable instead of stopping `status`, `diff`,
203
+ `accept --all` and the run's closing summary. The editor flags a
204
+ `{data:...}` reference only where the runner reads it (a step, its table
205
+ or doc string), never in a title, a description or a comment.
206
+ - A Then recorded again under the same words that now expects another
207
+ value ("Pending" where it checked "Paid", a status 500 where it expected
208
+ 201, another JSON value or URL), or reads another element named the same
209
+ way (test id "status" become "badge"), is held for review as a weakened
210
+ check (a row or data token holding the same value is the same): a stale re-record no longer replaces a check that just
211
+ failed. A heal that changed nothing is red, never a proposal, also with
212
+ `--no-verify`. The usage an agent session spent stays counted when its
213
+ proof replay cannot start.
214
+ - An Outline row's short value (a word under 5 characters with no digit,
215
+ such as "dev") is written as its `<column>` only where it stands alone:
216
+ inside "devices panel" it stays as recorded, so another row no longer
217
+ checks a text never on the page. A secret Playwright cut short with "…"
218
+ in a trace line, or a captured credential cut short in a message, is
219
+ masked.
220
+ - Replay: a "not shown" check looks inside the app's own iframes too (its
221
+ origin, or one it writes, shown on the page), as a "shown" check does, so
222
+ an error drawn in one no longer passes both; another origin's frame (a
223
+ chat widget's "Close") and a hidden one (a silent sign-in renewal) are
224
+ not searched. A "not shown", count, "differs" or "URL does
225
+ not contain" check, and a response check, passes only once the requests
226
+ the last action started have answered and the passing state has held for
227
+ half a second: a list still growing, a placeholder drawn during a
228
+ refresh or a row gone for a moment no longer passes. A long-poll the page
229
+ opened earlier is not waited for. A response check fails when the very
230
+ request it matched (its URL and method, and the body or JSON it checks)
231
+ was answered again with another status (a POST answered 200 and then
232
+ 500); a refetch, a child path or a GraphQL query on the same endpoint is
233
+ another request.
234
+ - `accept --step-set` refuses a StepSet that ran in a named browser session
235
+ (`As "seller"`) and a proposal that fails an Examples row: filed for
236
+ reuse, its steps would seed other sessions' or rows' steps. A Nordic
237
+ price ("150,- kr", "150:- kr") keeps its currency. A base URL's Basic
238
+ header an API echoes is masked; a URL in a query beside an email is not
239
+ taken for a password; step text accept writes into the feature file
240
+ keeps its URLs whole.
241
+ - Recording: a capture's target is read as replay reads it before it is
242
+ recorded, and a value it does not show is refused, so a later check never
243
+ compares with another element; a status the step names exactly ("answer
244
+ 404") is expected exactly, never as its class, and a status check names a
245
+ status the step states; a step performed again drops its requests'
246
+ responses too, so a later check never reads a dropped one; a response
247
+ check is verified with the step's table values, as replay reads it.
248
+ - Config: a key that names a known one misspelt (`maxCostUSD`, `stict`) is
249
+ refused, naming the key meant, and any other unknown key is said; a
250
+ `capabilities` value in quotes, an unknown `snapshotMode` and a
251
+ `hookTimeoutMs` of 0 are refused; an empty `SAFFRON_ENV` keeps the
252
+ config's `env`. `status --json` and the MCP `project_status` mask a
253
+ password in any URL and a header value written whole. `run --plan`
254
+ judges a pending proposal as the run does: a stale one is a recording,
255
+ one filed on another machine is not replayed, a row-bound one is
256
+ refused. The MCP `verify` tool says when the run did not finish cleanly
257
+ (interrupted, scenarios not run, a failed hook).
258
+ - A heal whose agent session could not start keeps the replay's failure,
259
+ never "not run"; a fixture that failed in the app fails the scenarios
260
+ behind it, never "environment missing"; the usage earlier agent
261
+ sessions of a recording spent stays counted when a later one fails.
262
+ - A password in the base URL, or a user name alone (`https://TOKEN@host`),
263
+ is masked in the report, the run history and bundles. A run with `--filter` or of part of the features folder is
264
+ partial in the history, so trends compare suite runs only. A red the
265
+ app did not cause never makes a scenario flaky on the dashboard.
266
+ - A CSV data file with a quote never closed is refused, naming the row,
267
+ instead of reading every line after it as one field. The language
268
+ server no longer stalls on a line holding a long run of spaces.
269
+ - Replay lets the first candidate that finds the recorded element decide a
270
+ check, shown or not: a hidden recorded element no longer passes a text,
271
+ visible, value or enabled check on a fallback elsewhere, a list check
272
+ (`every {data:...}`) no longer passes on a fallback or another frame
273
+ listing more, and a watched message counts only on the element watched,
274
+ never on a log line elsewhere. A currency-first value (`$5.00`) in a
275
+ pattern no longer matches its negative (`-$5.00`).
276
+ - A refused API request's reply to the agent masks a credential an API
277
+ handed out in the URL, as the reply to a request that answered does.
278
+ - Restoring a column over a secret no longer splices a short env value
279
+ into a word (`dev` in `/devices`); a row value an API request's path
280
+ holds URL-encoded is written back as its column, as in a browser URL.
281
+ - A red the agent cannot fix (an unreadable recording, a row refused, the
282
+ committed recording failing while another row's proposal waits) counts
283
+ as a failure in the report and the run summary, never as "needs
284
+ recording" with a command that would change nothing.
285
+ - The HTML report sets the reds the app did not cause apart from
286
+ failures: a stale recording in a run with `--no-agent`, a missing
287
+ environment, an agent that could not start or a run over its budget.
288
+ They get their own *not run* group under the failures (*needs
289
+ recording* when that is the only reason), a ○ on the run map, and a
290
+ *failed* count that says how many more did not run and why. Every
291
+ scenario that needs recording names the others, with one `saffron run`
292
+ command for their feature files. The run map counts them per feature
293
+ file, and the recurring failure themes leave them out (run history now
294
+ keeps the mark). They are still red, and the run still exits 1. Found in
295
+ real-world use, where one edit to a shared StepSet made 45 of 48 reds
296
+ stale recordings.
297
+ - A request group's step seeds another scenario that sends the same
298
+ requests with its own values: the same table header, request names and
299
+ rows, each cell read through `<row:column>`, and so does a group check
300
+ that reads each response's own row. Before, a group step was never
301
+ seeded, so every scenario recorded it with the agent.
302
+ - A StepSet accepted on its own (`accept --step-set`) after a scenario's
303
+ recording was made seeds the steps that come from that StepSet over the
304
+ scenario's older recording, with no `seeded from another recording:`
305
+ note; the same words written straight in a scenario keep its own.
306
+ - `saffron status --json` lists the StepSets each proposal recorded
307
+ (`stepSets`), for an editor to offer `accept --step-set`.
308
+ - `saffron eject` exports a count check as Playwright's count of shown
309
+ elements, `expect(locator.filter({ visible: true })).toHaveCount(n)`,
310
+ the number as replay reads it (`010` is 10).
311
+ - A pattern check that fails where the pattern writes an optional part
312
+ right after a value says the value is matched whole and how to write the
313
+ part instead.
314
+ - A currency code after a number counts as a price in a locator's name
315
+ only after two digits or more: "Section 4 SEK" is no price.
316
+
317
+ Editors (VS Code extension 0.8.1, JetBrains plugin 0.8.1)
318
+
319
+ - Both editors offer `saffron export` (the review bundle), a draft pull
320
+ request with the pending proposals (`export --branch`, after asking for
321
+ a branch and confirming), `saffron eject` (after asking, for the open
322
+ file or every one) and `accept --step-set` for one ticked proposal's
323
+ StepSet, from the Proposals view's menu (VS Code) or toolbar
324
+ (JetBrains).
325
+
8
326
  ## 1.3.3: Concurrent API requests, and recordings that follow their data
9
327
 
10
328
  Released 2026-11-05.