saffron-ai 1.3.1 → 1.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,267 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
5
5
  JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
6
6
  release heading names its main feature: `## <version>: <title>`.
7
7
 
8
+ ## 1.3.2: Exact checks for amounts, URLs and numbers, and secrets kept masked
9
+
10
+ Released 2026-11-04.
11
+
12
+ Runner
13
+
14
+ Upgrading from 1.3.1:
15
+
16
+ - A URL check (`expectUrl`, contains) reads its text as a whole part of
17
+ the URL, and a path (text with a `/`) outside the query, unless the text
18
+ names a query (`?`, `=`, `&`): `/dashboard` no longer holds on
19
+ `/login?next=%2Fdashboard`, nor `/orders/42` on `/orders/421`. A word
20
+ with no slash (`shoes`) is still found in `/search?q=red+shoes`, but not
21
+ in a query value that is a path (`dashboard` in `/login?next=/dashboard`),
22
+ nor in a query value whose encoded `#` or `&` makes it look like a
23
+ fragment or another field (`/dashboard` in `?next=%2F%23%2Fdashboard`).
24
+ A check that the URL does not hold a text still reads it anywhere:
25
+ `error` is in `?errorCode=5`. A text that names a query is the page's
26
+ own: its `?` is where the URL's query starts and a field it names is
27
+ one of the URL's own, so `/dashboard?tab=settings` and `tab=settings`
28
+ no longer hold on `/login?next=%2Fdashboard%3Ftab%3Dsettings`, nor
29
+ `/dashboard` on an OAuth callback's `#state=%2Fdashboard`. A fragment's
30
+ fields are the page's own too: `#tab=settings`, `/app#tab=settings` and
31
+ `tab=settings` hold on `/app#tab=settings`. A recording
32
+ that passed only that way turns red; the exported test checks the same.
33
+ - `Examples: {data: roles}` (a space, a capital `Data`) is an error naming
34
+ the file and line, where the Outline used to run no row at all. The
35
+ language server marks it while it is typed.
36
+ - An expected text never matches inside a longer word or number, whether
37
+ a row value or written as it is (SAF-197): `"$10"` no longer holds on
38
+ `$100`, `"1 item"` on `11 items`, nor a text target `Saved` on "You have
39
+ unsaved changes". A role's name may still have grown a suffix. A number
40
+ written in the step may be followed by a zero decimal part (`"$10"` in
41
+ `$10.00`, never `$10.50`, `₹1,00,000` or `2.0.1`); a row value keeps its
42
+ decimals whole (`<id>` 1 is not in `1.00`), and neither is found before
43
+ a digit in an inline tag (`Order 1<b>2</b>`, `$10<sup>.50</sup>`). A
44
+ text ending in a number's point or comma takes no more digits
45
+ (`"Refund of $10."` is not in `Refund of $10.50`), and a number is not
46
+ found in its negative (`"5.00"` is not in `-5.00` or `−5.00`; `"5"` is
47
+ still in `pages 3-5`). A value that starts or ends in Chinese, Japanese,
48
+ Thai, Lao, Khmer or Myanmar script is not bounded there, as those
49
+ scripts put no space between words, but for a negating prefix:
50
+ `"成功"` is not in `不成功` nor `没有成功`, `"保存"` not in `无法保存`.
51
+ Only a negation right before the value is read: a short value in a
52
+ sentence that negates it further back can still be found. A recording that passed only that way turns red, and the agent
53
+ records the whole word next time.
54
+ - An expected text never runs across a block, a table cell or a `<br>`
55
+ (SAF-198): `"15.00"` no longer holds on the cells `1` and `5.00`, nor
56
+ `"Qty 10"` on the items `Qty 1` and `0 left in stock`. Items of a flex
57
+ row or a grid read as one line, a space apart: `"Total: $10"` holds on
58
+ the items `Total:` and `$10`. A value written on several lines still
59
+ matches the lines the page draws.
60
+ - A style or attribute check needs its element shown (SAF-199): "the
61
+ error is shown in red" no longer holds on an error that is not on
62
+ screen, nor on a shown fallback when the recorded element is on the page
63
+ but hidden. A check of what hides an element (`display`, `visibility`,
64
+ `opacity`, `content-visibility`; `hidden`, `aria-hidden`, `inert`) still
65
+ holds on a hidden one, and so does one of a `class`, a `style`, `open`
66
+ or `data-state` attribute, or a height, width, transform or clip: how
67
+ many apps hide an element (`d-none`, a closed dialog).
68
+
69
+ The exported Playwright tests check text, blocks and styles the same way
70
+ (SAF-197, SAF-198, SAF-199).
71
+
72
+ - A text check of an amount written with its currency (`$5.00`, `€5`)
73
+ no longer holds on the negative amount (`-$5.00`, `−€5`), as `5.00`
74
+ already did not on `-5.00`. A URL check of a number takes no decimal
75
+ part (`amount=10` is not in `?amount=10.50`, nor `/orders/42` in
76
+ `/orders/42.5`), and a letter or digit the URL writes percent-encoded
77
+ belongs to the word beside it (`name=Jos` is not in `?name=Jos%C3%A9`,
78
+ nor `id=42` in `?id=42%35`). The exported test checks the same.
79
+
80
+ Fixed, from a review of the runner:
81
+
82
+ - A credential stays masked wherever it travels: a fixture's token that a
83
+ later capture replaced, or one inside an extracted object, in the
84
+ scenarios the fixture seeds; and a replaced token in what the agent
85
+ reads while recording, which it used to show. An exported test masks a
86
+ secret as a URL encodes it (`%2B`, `%2F`, `%3D`) and in the errors of
87
+ its JSON checks, as replay does.
88
+ - A `differs` check on an object holding an integer beyond JavaScript's
89
+ precision no longer passes against the same object a capture kept.
90
+ - The dashboard's suite health no longer reads 100% while a scenario
91
+ failed (473 of 475 reads 99.58%).
92
+ - The language server no longer marks `Examples: {data: roles}` as an
93
+ error inside a doc string, nor when a description comes before its
94
+ table: the runner reads both.
95
+
96
+ - No false green from a fallback selector or an iframe: the first locator
97
+ that finds a shown element decides a check, so a fallback that happens
98
+ to show the expected text no longer vouches for the recorded element
99
+ that failed it. A recorded element that is gone, or hidden for a check
100
+ other than a style or attribute check, still gives way to its fallback.
101
+ - No false green from an API check whose pattern matches any text: a
102
+ field, header or `until` pattern such as `.*`, `^` or an empty one is
103
+ refused while recording, as page checks already were.
104
+ - No Outline row passes on another row's data when the app writes a row
105
+ value without its accents (`José` as `status-jose`), or when a column
106
+ names a section without quotes (`the <plan> card shows "Popular"`).
107
+ - A captured id of one or two characters is recorded as its token where
108
+ a path segment or field names its capture (`/orders/42` for `order`), so
109
+ replay no longer reads the record made while recording. A field that
110
+ names another id (`productId: 1`) is left as it is.
111
+ - A secret in an API query that holds `/` together with `+`, `=` or `&`
112
+ (a base64 key) is masked in errors, transcripts, reports and the
113
+ agent's replies.
114
+ - A page or API message cut for a report is cut with `…`, so a secret
115
+ across the cut is masked too, also when the message carries it
116
+ URL-encoded.
117
+ - A request method taken from a column (`a <verb> request`) is kept as
118
+ sent, which binds the recording to its row, instead of an invalid
119
+ recording made again on every run.
120
+ - An integer above 2^53 extracted from a response (a snowflake id) keeps
121
+ the digits the API sent, instead of being rounded to another id. A JSON
122
+ check reads the same digits, so "no longer the same id" fails on the
123
+ same id, and a request naming the captured id still sends its token.
124
+ - A differs check finds the element again when the app draws it anew
125
+ (a React remount), where it used to fail for good.
126
+ - A shard run removes its own earlier report as it starts, so a shard
127
+ that then fails setup, crashes or is stopped is missing from
128
+ `report --merge` instead of standing in with the earlier run's results.
129
+ It also goes when the run stops before running anything (a feature file
130
+ error, a `--filter` that matches nothing).
131
+ - `saffron eject` warns when the project's own `playwright.config` records
132
+ a trace and the exported tests read secrets, `{env:...}` values or
133
+ credentials the API hands out, which Playwright's trace keeps unmasked
134
+ (SAF-200). It no longer only checked on the first eject, and it leaves
135
+ the file as it is. A config it writes leaves the trace out for
136
+ captured credentials too.
137
+ - A pending proposal stays with the Outline row whose values it was
138
+ recorded on when a row is inserted above it or the rows are reordered
139
+ (SAF-201), where another row used to take its "(example N)" and record
140
+ over it. `saffron status` and the plugins name that row as the rows
141
+ stand now, and the review page shows that row's pictures, where they
142
+ used to name and show the row that took its place.
143
+ - An API credential is masked wherever it could still show: a token a
144
+ polled response handed out before the one waited for, a token a later
145
+ capture replaced (a poll that rotates it, then quotes the old one as
146
+ `previous_token`), and an error raised while a request is built (a body
147
+ value that is no number quoting the captured token it resolves to). The
148
+ exported test learns a token from every polled response too, where it
149
+ learnt only the last one's and an error quoted the one before.
150
+ - A compiled `Then` keeps a quoted value as written: `the /name field is
151
+ "Ada Lovelace"` checks for both spaces, where it used to check for one.
152
+ - `saffron accept <file>#<revision> --step-set` refuses a proposal replaced
153
+ since it was listed, as a whole accept does, where it took the
154
+ replacement's steps unseen.
155
+ - A request sent several times at once waits for every copy when one gets
156
+ no response, and keeps the copies that answered in the transcript, where
157
+ the failure came first and a late write went unrecorded.
158
+ - An exported test extracts an integer beyond JavaScript's precision as the
159
+ digits the API sent, as replay does, where it rounded an int64 id into
160
+ another record's.
161
+ - Each API response is let go once read, in replay and in exported tests,
162
+ so a long poll no longer holds every response it read until the scenario
163
+ ends.
164
+ - An exported text check reads the copy replay reads, a shown one after a
165
+ hidden copy with the same text (a mobile menu's), where it read the
166
+ hidden one and failed.
167
+ - A step naming its column for the statuses of copies sent at once
168
+ (`refused with <refusal>`) records them as that column, as it records a
169
+ single status, so the other rows replay where they were refused as
170
+ recorded for one row.
171
+ - A JSON check reads a number JavaScript cannot hold as written as a
172
+ number, written as the API sent it: a `min` or `max` compares it exactly
173
+ with the bound as written, in any notation (`9007199254740993`,
174
+ `9007199254740993.0` and `9.007199254740993e15` are all over a maximum
175
+ of `9007199254740992`, `9007199254740991.1` over one of
176
+ `9007199254740991`, and `1e9999999999` over `1`, where it stopped the
177
+ run), and a `pattern` (which reads text) does not hold on it. An `equals` still checks its digits, as the error for a typed
178
+ number advises.
179
+ - A compiled `Then` leaves a number JSON cannot hold (`the /n field is
180
+ 007`) to the agent, where it compiled a check that could never pass.
181
+
182
+ Fixed, from real-world use of 1.3.1:
183
+
184
+ - The report's pass rate never reads 100% while a scenario failed: 473 of
185
+ 475 reads 99.58%, where it was rounded up to 100%. The dashboard's
186
+ suite health, recorded and tag shares read the same way, and 0% means
187
+ none. `passRate` in
188
+ `saffron status --json` and the history stays a whole number, now never
189
+ 100 unless every scenario passed (99 for 473 of 475).
190
+
191
+ - An Outline row that fails in a step before any step uses its own values
192
+ (a lunch sold out in a shared checkout step) is blamed on the app's
193
+ state, blocked by the environment, where the report used to call it
194
+ the recording's fault and point at a column the step never read. A
195
+ value used in a step's data table or doc string counts as used.
196
+ - A pending proposal stays with its row when a column is added to the
197
+ Examples: only the columns its row had tell rows apart, where every row
198
+ was called another row's and none recorded again.
199
+ - The sensitivity check judges a check against the action before a step
200
+ that only reads (a captured value, a watch armed), as it already did
201
+ for a wait: reading a booking number is not what confirms a payment.
202
+ - `api_contract` (the MCP tool) and the agent's contract summary list a
203
+ request body's alternatives with their fields (`one of: { ... } | { ...
204
+ }`) and an `allOf`'s parts as one body, where they said `any`.
205
+ - The Saffron skill's syntax reference describes the negative and exact
206
+ URL checks 1.3.1 added, where it said there were none.
207
+ - A recorded API step no longer keeps the agent's failed attempts
208
+ (SAF-202). A request whose body the pinned contract refuses is not
209
+ recorded, even when the API answered 2xx, unless the step expects a
210
+ refusal. The body is judged as sent, its tokens resolved: `{date+1}` as
211
+ a date, and `{captured:id}` as the text it sends, so an integer field
212
+ takes the number itself. A
213
+ request sent again to the same operation and expected answer right
214
+ after the step's last one takes its place, unless the step asks for
215
+ several ("twice", "two customers", "for each", "once more"), the earlier
216
+ one sent a value the step names that this one does not ("the currency
217
+ to SEK and guest names on" is two writes), this one sends the same
218
+ fields with other values the step does not name ("a mug and a hat"), or
219
+ both carry the same idempotency key. A value counts as named in any
220
+ case and in the step's data table or doc string, and as sent in a form
221
+ or the query too (`?status=draft`, then `?status=archived`); a step
222
+ with a table of several rows keeps every request.
223
+ - A `Then` that names a section in plain words (`the confirmation Summary
224
+ shows the package`, `the order card shows Paid`) has its checks scoped
225
+ to that section, as a quoted name already did (SAF-203): a check that
226
+ reads the whole page is refused while recording, instead of passing on
227
+ text that was there before the action.
228
+ - An Outline whose recorded row held one value in two columns
229
+ (`currency` and `display` both `NOK`) no longer refuses the other rows
230
+ for a step that names only one of them (`I choose "<currency>"`): the
231
+ recorder writes the column the step names. A step naming both, or
232
+ neither, still records again on a row that types them apart.
233
+ - A request that sends its header's credential in the body or URL as well
234
+ (`Authorization: Bearer {env:KEY}` and a body field `apiKey`) is refused
235
+ while recording, unless the step names both the body and the header.
236
+ Only an auth header counts: `X-Auth-Token`, not `X-Author` or
237
+ `X-Auth-User`.
238
+ - A "not shown" check naming a control by the start of its name (a
239
+ textbox `Firstname`, where the page shows `Firstname Required`) gets an
240
+ insensitive check note saying no element has that name, and naming the
241
+ control shown before the step.
242
+ - A check that finds its element by the text it checks (an amount found as
243
+ the text `SEK 495.00` anywhere on the page) is refused while recording:
244
+ it holds wherever that text shows. The agent finds the field the step
245
+ names, or records a visible check when the step only says the text is
246
+ shown.
247
+
248
+ Docs
249
+
250
+ - **How a recording is checked.** A new section of the guide (§24), and a
251
+ page of its own on the site, maps every check Saffron runs on a
252
+ recording before it is filed, in order: what each note means, which ones
253
+ hold a proposal from `accept --all` and when to accept it by file, what
254
+ is on by default, and how to turn on stability proofs.
255
+
256
+ Editors (VS Code extension 0.7.8, JetBrains plugin 0.7.9)
257
+
258
+ - Both editors mark `Examples: {data: roles}` or `{Data:roles}` with no
259
+ table under it as an error where it is written, as the runner stops on
260
+ it: VS Code in every project, JetBrains through the runner's language
261
+ server. Neither marks one inside a doc string or with a description
262
+ before its table.
263
+ - VS Code: the "Proposals pending review" row asks before it runs
264
+ `saffron accept --all`.
265
+ - JetBrains: Accept All and an accept or reject of selected proposals no
266
+ longer become the selected run configuration, so the Run button never
267
+ repeats them.
268
+
8
269
  ## 1.3.1: Checks for what vanishes, URL and style checks, and a StepSet accepted on its own
9
270
 
10
271
  Released 2026-11-02.