saffron-ai 1.3.2 → 1.3.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,164 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
5
5
  JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
6
6
  release heading names its main feature: `## <version>: <title>`.
7
7
 
8
+ ## 1.3.3: Concurrent API requests, and recordings that follow their data
9
+
10
+ Released 2026-11-05.
11
+
12
+ Runner
13
+
14
+ Upgrading from 1.3.2:
15
+
16
+ - Recordings with a request group (several API requests sent at once) are
17
+ schema version 19, and recordings with a count check use an action
18
+ 1.3.2 does not know: 1.3.2 refuses both rather than replaying them
19
+ wrong. Upgrade every machine that replays them, CI included.
20
+ - Two new notes hold a proposal from `accept --all`: `check weakened:`
21
+ (a `Then` recorded again with a looser check than the committed one)
22
+ and `data not followed:` (a step that names a `{data:...}` value its
23
+ recording keeps as text). Accept such a proposal by its file after
24
+ reading why.
25
+ - While recording, each browser check the agent writes is run on its page
26
+ by replay's own engine, and refused when it does not hold there. A
27
+ recording that used to fail its proof on such a check now fixes it
28
+ before the proof.
29
+ - A scenario recorded again because its recording is stale seeds the
30
+ steps it did not change from its own recording, never from another
31
+ scenario's newer one.
32
+
33
+ - A downloaded file's check resolves a `{captured:...}` value, while
34
+ recording and at replay, so "the calendar file contains the booking
35
+ number `{captured:bookingNumber}`" checks this run's booking; one that
36
+ nothing captured fails saying so. Before, replay checked for the token
37
+ as text and the proof failed every file.
38
+ - A date the scenario writes as a token stays that token when the agent
39
+ types it glued to a time (`20261111T140000Z` for `{date+7:YYYYMMDD}`),
40
+ and a pattern that writes the time after it (`{date+7:YYYYMMDD}T[0-9]{6}`)
41
+ matches it. A time written as optional (`(?:T[0-9]{6})?`) still ends the
42
+ date, so the step writes it as required. A date left as text
43
+ in the format the scenario's own token writes gets `date not following
44
+ the clock:` and is held from `accept --all`.
45
+ - A short data value the step names (`{data:party.size}`, 2) is written
46
+ back as its reference where it stands alone once in a check's or fill's
47
+ value, a text pattern or an amount ("Table for 2 people", `"price" *
48
+ 2`), never in a selector, URL or path, nor a digit of regex syntax, a
49
+ time or a date. A step whose recording still holds a value it
50
+ names as text, or a list's items one by one, gets a new `data not
51
+ followed:` note and is held from `accept --all`. A price in a locator's
52
+ name (`Info Lunch SEK 150`) is a `volatile locator:`.
53
+ - The language server sees a StepSet's steps filed with `accept
54
+ --step-set` (`.saffron/step-sets`) as recorded when they change.
55
+ - The plugins' keyword tables carry each language's Feature, Background
56
+ and Rule, so a file in another Gherkin language ends a scenario at them.
57
+ - A `StepSet:` or `Fixture:` in a file written in another Gherkin
58
+ language (`# language: de`) defines its set or fixture: it was read as
59
+ the English `Scenario:`, which that language does not know, so no set
60
+ was found. A StepSet is called in a language with no `*` step too
61
+ (Slovenian), and the `# language:` line is read as Gherkin reads it:
62
+ `# language: de # Deutsch` is English to both.
63
+ - The language server reads an `As "..."` session through Background and
64
+ Rule headers in the file's own language, and ends one at a `Fixture:`,
65
+ as the VS Code extension does.
66
+ - A file the agent uploads while recording is found in the project when
67
+ its path is relative, wherever the run was started: `saffron -p app
68
+ run` from a monorepo's root refused `demo-app/sample-upload.txt` as
69
+ outside the project.
70
+ - The benchmark's recording measure counts every note held from `accept
71
+ --all`, and names the scenarios not verified on the first attempt or
72
+ held, with why.
73
+ - A browser check the agent writes is run on its page by replay's own
74
+ engine before it is recorded, with replay's values, captures and clock,
75
+ and refused with replay's reason when it does not hold: a value not
76
+ whole in its text or an element not shown fails while recording, not
77
+ at the proof.
78
+ - A `Then` recorded again under the same words with a looser check than
79
+ the committed one gets a `check weakened:` note and is held from
80
+ `accept --all`: a value it no longer reads (the same value written
81
+ another way still counts), a pattern's or name pattern's lost `^` or
82
+ `$`, a text it checks less of, a kind of check it no longer makes or
83
+ covers, a JSON field it no longer compares, an element it no longer
84
+ names. `--rerecord` keeps the recording it removes for this comparison.
85
+ - A scenario recorded again because its recording is stale seeds each
86
+ step it did not change from its own committed recording where that fits
87
+ and the page shows what it acts on, never from another scenario's newer
88
+ one (a shared StepSet's update brought those in). A step that replayed
89
+ another scenario's recording is named in a `seeded from another
90
+ recording:` note.
91
+ - A check of an element that runs out of time (shown, text, pattern,
92
+ value, style, attribute) says what replay saw last: nothing matching
93
+ the target, an element there but not shown, or the text or value it
94
+ showed, with secrets masked before it is cut.
95
+ - A count check: "3 rooms should be listed" is one check of how many
96
+ shown elements the target finds in the page (or the frame it names),
97
+ from a number, a `<column>` or a `{data:...}` value; a hidden element
98
+ never counts, and none is written as "not shown". `saffron eject` does
99
+ not export it yet, and saffron-ai 1.3.2 refuses a recording with one as
100
+ an action it does not know, never replaying it wrong.
101
+ - The report page's headline counts and cost are this run's own after a
102
+ run of named scenarios, with the number of rows kept from earlier runs
103
+ beside them; before, it added up the kept rows' results and AI calls.
104
+
105
+ - Several API requests at once, each with its own values, and checks over
106
+ all their responses: an idempotency test that registers one charge from
107
+ several requests at the same moment. A step's table names the requests,
108
+ a header row and a row per request (`When I send these registrations
109
+ concurrently as "registrations":`); the request is recorded once and
110
+ sent once per row, every row before any answers, each row's cell
111
+ written as `<row:column>`, so editing a cell needs no recording again;
112
+ a `<row:column>` naming no column of the table is refused rather than
113
+ sent as text, and the group's name may come from an Examples column.
114
+ Every response stays with its row whatever order it arrives in, a
115
+ request that gets none is kept and fails the step once all have
116
+ settled, and a group is never sent again. `Then` steps check every
117
+ response, those of one status, a named one, at least some, or all of
118
+ them sharing a field's value (`all 200 responses should have the same
119
+ "/paymentReference"`); a response's own cell in a check is its row's.
120
+ The common checks are exact phrases recorded with no agent. A failure
121
+ lists each request with its status and why it failed. "The response"
122
+ after a group fails instead of reading one of them. Recordings with a
123
+ group are schema version 19: saffron-ai 1.3.2 and earlier refuse them.
124
+ `saffron eject` does not export a group yet.
125
+
126
+ - `saffron accept --propagate` fixes a healed API request everywhere it
127
+ is sent, as it already did for a locator. Scenarios that share a step's
128
+ wording hold copies of its request; when a heal rewrites that request
129
+ (a field the API now requires, a renamed path), accepting it with
130
+ `--propagate` rewrites every other cache that sends the same request
131
+ under the same step text, so one heal fixes them all instead of each
132
+ scenario healing on its own run. `--propagate` goes on the accept itself:
133
+ without it, accept names the caches it left as they are, which heal on
134
+ their own run. Either way it prints what the request was and is now. A
135
+ fix is not copied when it writes a value of the healed scenario's own (a
136
+ table cell, a doc string, its Outline row), adds a `<column>` or
137
+ `{captured:...}` value, drops a value the request extracted, sends the
138
+ request to another API, or when the heal kept the request as it was in
139
+ one place and rewrote it in another; accept says which. A step whose
140
+ heal added or removed an action is not copied either.
141
+ - `saffron accept` without `--propagate` no longer tells you to re-run it
142
+ with `--propagate`: the proposal is accepted by then, and a re-run had
143
+ nothing left to fix. It says the other caches were left to heal on
144
+ their own run.
145
+
146
+ Editors (VS Code extension 0.8.0, JetBrains plugin 0.8.0)
147
+
148
+ - Both editors mark a proposal with a `check weakened:` or `data not
149
+ followed:` note, or one that fails an Examples row, as held from Accept
150
+ All, as `saffron accept --all` skips it. JetBrains asks before Accept
151
+ All runs, as VS Code does.
152
+ - In a file written in another Gherkin language (`# language: de`),
153
+ Right-click Run reads that language's Feature, Background and Rule
154
+ headers (JetBrains), and VS Code's completion, recording badges, hovers
155
+ and wording hints read its step keywords and headers, a keyword with no
156
+ space of its own ("假如") included.
157
+ - A `Fixture:` block is a block of its own: it ends the StepSet above it
158
+ (VS Code), and Right-click Run inside it runs no scenario (JetBrains).
159
+ - VS Code shows the steps a StepSet's proposal filed for reuse
160
+ (`.saffron/step-sets`) as recorded, and no longer marks `Examples:
161
+ {data: roles}` as an error when a description line and then a table
162
+ follow it.
163
+ - JetBrains passes a run's files after `--`, so a file whose name starts
164
+ with a dash is a file, never an option.
165
+
8
166
  ## 1.3.2: Exact checks for amounts, URLs and numbers, and secrets kept masked
9
167
 
10
168
  Released 2026-11-04.