saffron-ai 1.2.1 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +419 -0
- package/dist-pkg/cli.js +845 -459
- package/package.json +4 -3
- package/skills/saffron/SKILL.md +32 -3
- package/skills/saffron/references/api.md +15 -0
- package/skills/saffron/references/config.md +1 -1
- package/templates/playwright/saffron-helpers.ts +58 -7
- package/textmate/saffron/syntaxes/saffron.tmLanguage.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,425 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 1.3.0: Recordings checked by machine, and a review every view agrees on
|
|
9
|
+
|
|
10
|
+
Released 2026-10-30.
|
|
11
|
+
|
|
12
|
+
Runner
|
|
13
|
+
|
|
14
|
+
Upgrading from 1.2.2:
|
|
15
|
+
|
|
16
|
+
- A recording that keeps part of a captured text (`pattern`) is version
|
|
17
|
+
13, which 1.2.2 and earlier refuse ("Update saffron-ai"). Upgrade every
|
|
18
|
+
machine and CI job that replays the suite together.
|
|
19
|
+
- `accept --all` now also holds proposals with a `row not proved:`, `date
|
|
20
|
+
not following the clock:`, `write not finished:` or `check changed the
|
|
21
|
+
page:` note: accept them by file once reviewed. A teammate still on
|
|
22
|
+
1.2.2 running `accept --all` takes them, so upgrade before
|
|
23
|
+
bulk-accepting. An Outline recording that fails a row it serves is filed
|
|
24
|
+
unverified, its proof error naming the row, so every runner skips it.
|
|
25
|
+
- Now refused: an Examples table that names a column twice; a text check
|
|
26
|
+
a step words as a comparison ("starts with") but records as
|
|
27
|
+
containment; under a `Then`, an action that changes the app (it is not
|
|
28
|
+
recorded, and the proposal is held); a captured credential (a login's
|
|
29
|
+
token) typed through the browser.
|
|
30
|
+
- `transport.api.headers` go only to an API under test; an unset
|
|
31
|
+
`{env:NAME}` in them, or a value that is not text, stops the request
|
|
32
|
+
instead of sending it.
|
|
33
|
+
- The run history (`.saffron/history.jsonl`) keeps its newest half once
|
|
34
|
+
it passes 64 MB.
|
|
35
|
+
- `saffron eject` refuses a capture that keeps part of its text
|
|
36
|
+
(`pattern`), which an exported test does not do yet.
|
|
37
|
+
- `--plan` counts an Outline as one recording. New output fields:
|
|
38
|
+
`estimate` (`--plan --json`), `failedRows` (`status --json`, the MCP
|
|
39
|
+
`verify` tool, the CI run view), `rows` (`diff --json`, the report
|
|
40
|
+
JSON), `invocation` (the report), `checkedAt` (a recording), and
|
|
41
|
+
`proof-N-requests.json` evidence files.
|
|
42
|
+
|
|
43
|
+
- **Recordings checked by machine, not read by people.**
|
|
44
|
+
- An Outline's recording is proved on every other row it serves, at
|
|
45
|
+
zero AI, in a fresh execution each, before it is filed. One that fails
|
|
46
|
+
on a row is filed unverified, its proof error naming the row, with a
|
|
47
|
+
`row not proved:` note; one that cannot serve a row (bound to its own
|
|
48
|
+
row's values) gets the same note, as accepted that row stays red until
|
|
49
|
+
it is recorded again. Both are held from `accept --all`. Rows whose
|
|
50
|
+
`@requires:` capability this environment lacks are not proved, and the
|
|
51
|
+
review counts them apart: "proved on 2 of 2 rows (1 refused)".
|
|
52
|
+
(SAF-159)
|
|
53
|
+
- A new recording is checked against a moved clock: every `{date}` token
|
|
54
|
+
resolved for tomorrow, the month's last day and the next month's
|
|
55
|
+
first, and the year's last day and the next year's first, and a
|
|
56
|
+
weekday, month or year left as text beside it must still match
|
|
57
|
+
(`Thursday, {date+1:MMMM D, YYYY}` does not; an amount such as
|
|
58
|
+
`2026.00` is no year). A whole date near today left as text is named
|
|
59
|
+
too, in the locale's own spellings and in the other order where only
|
|
60
|
+
one reading is a day ("13/11/2026"), unless the scenario writes it or
|
|
61
|
+
it is part of an address (`/blog/2026/10/30/`).
|
|
62
|
+
Either gets a `date not following the clock:` note, held from
|
|
63
|
+
`accept --all`. (SAF-158, SAF-165)
|
|
64
|
+
- A proof that passed while a write to the app's own site was cut off by
|
|
65
|
+
its frame moving on gets a `write not finished:` note, held from
|
|
66
|
+
`accept --all`; a replay that fails after such a write names it. The
|
|
67
|
+
app's own writes are those to the base URL, an API the run tests, or
|
|
68
|
+
the page's own site; an iframe loading, a third party's beacon, a
|
|
69
|
+
download or a 204 count for nothing, nor a read sent as a POST (a
|
|
70
|
+
GraphQL query, a search), judged by the rule a `Then`'s requests are;
|
|
71
|
+
a navigation that fails on the network counts. (SAF-170)
|
|
72
|
+
- A text check that passed on an element showing far more than the value
|
|
73
|
+
(a card) gets a `broad check:` note. (SAF-171)
|
|
74
|
+
- A page check is judged against the last step that acted on the page:
|
|
75
|
+
an API request in between cannot change the page, and named an
|
|
76
|
+
insensitive check that was not.
|
|
77
|
+
- A check of an Examples cell that is a day written out (`2026-01-05`,
|
|
78
|
+
`05-01-2026`, "Jan 5", "5th January 2026"), shown in another format
|
|
79
|
+
("05.01.2026"), binds the recording to its row, as a `{date}` cell
|
|
80
|
+
does: every row checked row 1's day. A cell such as "Order 1234" is no
|
|
81
|
+
day.
|
|
82
|
+
- `rowProofs` in saffron.config.json caps how many other rows a
|
|
83
|
+
recording is proved on; a row past it is "not run", and replays the
|
|
84
|
+
proposal on its own row later in the run.
|
|
85
|
+
- A contract re-check (only a pinned API contract changed) files the
|
|
86
|
+
recording with the Examples row, bindings, data and recording time it
|
|
87
|
+
was made with, whichever row ran the check, and records when it was
|
|
88
|
+
proved again as `checkedAt`. A re-check that would change any action
|
|
89
|
+
is red, never filed. (SAF-157)
|
|
90
|
+
- A stale reason and a row refusal start with the file and line they
|
|
91
|
+
point at (`features/plans.feature:5 step 2 reads ...`), and a held
|
|
92
|
+
note names its step's file and line after its kind. (SAF-160)
|
|
93
|
+
- **What a recording keeps.**
|
|
94
|
+
- The agent types the scenario's references (an Examples `<column>`, a
|
|
95
|
+
`{data:...}` value, a `{captured:name}` an earlier step kept) as
|
|
96
|
+
written: the browser gets the value and the recording keeps what the
|
|
97
|
+
agent wrote, as with `{env:...}`. Two references that read the same in
|
|
98
|
+
one call are left to the recording's own row matching. A captured
|
|
99
|
+
credential is never typed through the browser: the agent would read
|
|
100
|
+
it back, and the page could send it anywhere. (SAF-161, SAF-166)
|
|
101
|
+
- A value captured while recording can be typed by a later step
|
|
102
|
+
(`saffron_capture`, then `{captured:name}`). A capture can keep only
|
|
103
|
+
what a pattern picks (one group, named or not; it may hold
|
|
104
|
+
`{captured:...}`, `{date...}` and `<column>` tokens), so a booking
|
|
105
|
+
number beside its Copy label is the number alone; replay fails naming
|
|
106
|
+
the text when the pattern picks no value or two, or names one that
|
|
107
|
+
backtracks (`(a+)+`), which is stopped instead of freezing the run, as
|
|
108
|
+
a `matches` check's pattern is, and a `differs` check
|
|
109
|
+
against it compares the same part of the text. Such a recording is
|
|
110
|
+
version 13. A value the proof read with more around it (a label) gets
|
|
111
|
+
a `capture differs:` note. (SAF-166)
|
|
112
|
+
- A day the scenario writes with its month's name and no year
|
|
113
|
+
(`{date+9:D MMM}`) goes back to its token in every step, as a whole
|
|
114
|
+
date does, and so do the spellings "Nov 6 2026", "6. November 2026",
|
|
115
|
+
"2026/11/06" and, in the United States, "11/6/2026"; a path's segments
|
|
116
|
+
(`/blog/2026/10/30/`) stay as they are. (SAF-165)
|
|
117
|
+
- A `Then` checks; it never changes the app. An entry, a key press, a
|
|
118
|
+
pick, a tick, an upload, a drag, a navigation, a tab opened at an
|
|
119
|
+
address or a request that writes is not recorded under it: a write
|
|
120
|
+
request is refused before it is sent, and the proposal gets a `check
|
|
121
|
+
changed the page:` note, held from `accept --all`. A read the API
|
|
122
|
+
offers only as a POST may be sent: a GraphQL query to a `/graphql`
|
|
123
|
+
endpoint that holds no mutation (comments and strings aside), or a
|
|
124
|
+
request to an endpoint whose last segment is `search` or `query`. A
|
|
125
|
+
click or an answered dialog under a `Then` is recorded, as it may only
|
|
126
|
+
open what is checked, and held the same way, as nothing the run sees
|
|
127
|
+
can tell an opening click from one that cancels the order the check
|
|
128
|
+
reads: accept it by file once you have seen it only opens. (SAF-167)
|
|
129
|
+
- A step performed again after an attempt that did not work drops the
|
|
130
|
+
earlier attempt (`saffron_step restart`). (SAF-167)
|
|
131
|
+
- A text check the step names as a comparison (`starts with "Oct"`,
|
|
132
|
+
`ends with "-2026"`, "the same day as") is refused as plain
|
|
133
|
+
containment, naming the check that records it, in straight or curly
|
|
134
|
+
quotes or as a capitalized word (`starts with Oct`), and a pattern
|
|
135
|
+
recorded for it needs its anchor, in any group (`(?<id>^ORD\d+)`,
|
|
136
|
+
`(?i:^ORD)`) or as a lookahead (`(?=$)`). A number with a leading zero
|
|
137
|
+
("0042") is an identifier, never compared as an amount. (SAF-169)
|
|
138
|
+
- A choice opened through an element named with the row's `<column>`
|
|
139
|
+
gets a `choice opener:` note, and the agent is told to open a choice by
|
|
140
|
+
its own label, never by the value it shows. (SAF-168)
|
|
141
|
+
- **Review: every view says the same.**
|
|
142
|
+
- The review shows an Outline recording's proof row by row. One that
|
|
143
|
+
failed a row is never shown as verified: in `saffron run`, its totals
|
|
144
|
+
and cost split, the report, history and the dashboard, `saffron diff`,
|
|
145
|
+
`status --json` and the MCP tools (`failedRows`), the CI run view, the
|
|
146
|
+
pull request body and `saffron trace`, which asks before accepting it.
|
|
147
|
+
A later run that replays it keeps what its own proof found. (SAF-172)
|
|
148
|
+
- A held note stands beside the step it names in `saffron diff` and the
|
|
149
|
+
report's review tab, its file and line included, and the pull request
|
|
150
|
+
body names every note `accept --all` holds.
|
|
151
|
+
- A repair that fixes the steps before a `Then` while the `Then` still
|
|
152
|
+
fails as written, or that changes nothing and still does not replay,
|
|
153
|
+
is red, never a proposal healed again on every run.
|
|
154
|
+
- **Replay.**
|
|
155
|
+
- `ControlOrMeta`, in a key press or a click's modifiers, replays as the
|
|
156
|
+
browser's own platform has it, so a remote Linux browser driven from a
|
|
157
|
+
Mac pastes too. (SAF-175)
|
|
158
|
+
- A step on a page busy in its own script times out as it should.
|
|
159
|
+
- A popup no longer crashes the run: its first navigation has no frame
|
|
160
|
+
yet, which the network listener now allows for.
|
|
161
|
+
- A refinement after a failed proof gets the values the proof captured.
|
|
162
|
+
- **API tests.**
|
|
163
|
+
- The transport's API headers go only to an API under test, never to
|
|
164
|
+
another origin a request names. Each `{env:NAME}` is read when the
|
|
165
|
+
request is sent, and one that is not set is refused instead of sent
|
|
166
|
+
empty. A value read from `{env:...}`, or written whole for a header
|
|
167
|
+
whose name ends in a credential's word, a version aside
|
|
168
|
+
(`Authorization`, `X-Api-Key`, `X-Session-Id`, `X-Hub-Signature-256`,
|
|
169
|
+
`X-API`), is masked wherever it would show, including a response the
|
|
170
|
+
agent reads. A browser scenario's API requests carry the headers too,
|
|
171
|
+
masked in its trace, leaving out one whose `{env:...}` the job does not
|
|
172
|
+
set.
|
|
173
|
+
- A credential inside an extracted object (`extract "/data"` holding a
|
|
174
|
+
token) makes the capture a credential, and the token is masked on its
|
|
175
|
+
own, in a step that fails too. A name counts by its last word
|
|
176
|
+
(`accessToken`, `authorization`, `pwd`, `otp`, `api_key`,
|
|
177
|
+
`refresh_token`), a word that only says how it is held aside
|
|
178
|
+
(`authCode`, `apiKeyValue`, `privateKeyPem`), never one that only
|
|
179
|
+
mentions it (`sessionTitle`, `pinColor`, `total_tokens`): one rule for
|
|
180
|
+
captures, fields and headers, in replay and in an exported test alike.
|
|
181
|
+
A value of 8 characters or more counts, a password at any length, or a
|
|
182
|
+
code of 4 to 8 digits under `otp` or `pin`; a short word
|
|
183
|
+
(`"auth": "yes"`) is no secret, and masked it garbled every text it
|
|
184
|
+
appeared in.
|
|
185
|
+
- A secret of any length is masked: a value of some 7,000 characters
|
|
186
|
+
(a large token or cookie) made masking throw.
|
|
187
|
+
- A proof's API transcript is kept apart from the run's
|
|
188
|
+
(`proof-1-requests.json`), for API and browser scenarios alike. The
|
|
189
|
+
report, `saffron trace` and the CI run view show the run's own
|
|
190
|
+
requests first, and each proof's under its own name and outcome.
|
|
191
|
+
- Copies sent at once match their expected statuses in any order.
|
|
192
|
+
- **CLI, plan and reports.**
|
|
193
|
+
- A run of named scenarios folded into the last report carries its own
|
|
194
|
+
command, and says what it ran as `invocation`. (SAF-164)
|
|
195
|
+
- `--plan` estimates the browser time of its replays and recordings
|
|
196
|
+
from past runs, counts an Outline as one recording with its row
|
|
197
|
+
proofs, and counts no recording time with `--no-agent`. It counts a
|
|
198
|
+
new step's scenarios per file, and mentions `reset` as it is set.
|
|
199
|
+
(SAF-174)
|
|
200
|
+
- A suggested rename to an existing wording needs the two wordings to
|
|
201
|
+
share a word or a quoted argument, which it names, in a recording and
|
|
202
|
+
in `status` alike. (SAF-173)
|
|
203
|
+
- An Examples table that names a column twice is refused with its file
|
|
204
|
+
and line, and a file whose only Outline has no Examples rows is told
|
|
205
|
+
it needs one. While one file does not parse, the step vocabulary (the
|
|
206
|
+
LSP, MCP `search_steps`, `saffron vocab`) still offers every other
|
|
207
|
+
file's steps, and names the file that does not.
|
|
208
|
+
- `report --merge` refuses a shard whose rows or proof transcripts it
|
|
209
|
+
cannot read, a history line that is not a run is skipped, and the
|
|
210
|
+
dashboard calls a scenario flaky only when it flips in one
|
|
211
|
+
environment.
|
|
212
|
+
- The run history is read from its end, so only the runs a view needs
|
|
213
|
+
are read, and it keeps its newest half once it passes 64 MB: months of
|
|
214
|
+
CI runs made every command fail. A history that cannot be read costs a
|
|
215
|
+
run its trends, never its report.
|
|
216
|
+
- A recording with many dates is finished several times faster.
|
|
217
|
+
- `report --no-open`, and `report --run --no-open`, write the report
|
|
218
|
+
without opening a browser.
|
|
219
|
+
- `saffron run --no-agent` names the pending proposal that does not
|
|
220
|
+
replay on a row, instead of saying no recording exists, and the run
|
|
221
|
+
line no longer hides a path behind an identifier named like a secret
|
|
222
|
+
(`..._SESSION_ID`). An invalid `rowProofs` stops `--plan` too.
|
|
223
|
+
- The MCP `verify` tool takes `baseRef` only as a git ref, and says so
|
|
224
|
+
plainly otherwise; `saffron lsp --stdio` is accepted; `saffron prune`
|
|
225
|
+
says its count holds recordings, proposals and evidence folders.
|
|
226
|
+
- **Data tables for API steps.** An API request's values in a data table
|
|
227
|
+
under its step (an API key, a correlation id, query parameters, body
|
|
228
|
+
fields) record as their `<table:key>` wherever the request sends them,
|
|
229
|
+
path, headers, body or form, so editing a cell replays at zero AI. The
|
|
230
|
+
bundled skill and `saffron author` now tell the agent to keep each step
|
|
231
|
+
one sentence and put more than two values in such a table, not on one
|
|
232
|
+
long line.
|
|
233
|
+
|
|
234
|
+
Editors (VS Code extension 0.7.6, JetBrains plugin 0.7.7)
|
|
235
|
+
|
|
236
|
+
- Proposals with every new note are held, the Accept All tooltips name
|
|
237
|
+
them all, and an Outline proposal that failed a row shows as failing
|
|
238
|
+
it, in the proposal list and the CI run view.
|
|
239
|
+
- A `.saffron` file's `Fixture:` line is coloured as a block keyword, as
|
|
240
|
+
`Scenario:` is, in the editors and in the TextMate bundle the runner
|
|
241
|
+
ships.
|
|
242
|
+
|
|
243
|
+
## 1.2.2: Triage in the report, a new dashboard, and verification for coding agents
|
|
244
|
+
|
|
245
|
+
Released 2026-10-28.
|
|
246
|
+
|
|
247
|
+
Runner
|
|
248
|
+
|
|
249
|
+
- **Proposals are reviewed in the run report.** A scenario with a proposal
|
|
250
|
+
has a `run | review` tab pair. The review tab holds what used to sit
|
|
251
|
+
among the run's panels: every step beside what it records today and what
|
|
252
|
+
is proposed (unchanged steps dimmed), the notes on a step, the agent's
|
|
253
|
+
account, the page before the agent took over, the proof and stability
|
|
254
|
+
proofs, and the commands to accept or reject, each with a copy button.
|
|
255
|
+
The header has a "N proposals to review" chip that opens the first one.
|
|
256
|
+
The page before the agent took over is embedded once, as before.
|
|
257
|
+
`saffron diff --html` is unchanged. The report stays read only.
|
|
258
|
+
- **A new dashboard, in both editors and `saffron dashboard`.** The page
|
|
259
|
+
opens on the suite's health and a history grid: each scenario that is
|
|
260
|
+
failing, flaky or healed again and again gets one cell per run (last 20),
|
|
261
|
+
so a new break, a flaky test and a drifting page read at a glance. Click
|
|
262
|
+
a row for its error, and a scenario's name to open it in the editor.
|
|
263
|
+
A scenario that fits several chips (failing and flaky, say) is listed
|
|
264
|
+
once, under the most urgent group, and is found under each chip; a
|
|
265
|
+
group's count follows the chip, and a chip with nothing under it cannot
|
|
266
|
+
be pressed. The grid's columns are the last 20 runs a listed
|
|
267
|
+
scenario ran in, so runs from an editor's Run button of other scenarios
|
|
268
|
+
no longer stretch it with empty columns.
|
|
269
|
+
Slowest, tags and coverage gaps sit beside it. The pass-rate and
|
|
270
|
+
execution-time bar charts are gone. The editors need no update: they show
|
|
271
|
+
the runner's page.
|
|
272
|
+
- **Found in a wider test pass.**
|
|
273
|
+
- A day the page spells with its weekday ("Thursday, October 29,
|
|
274
|
+
2026") in a scenario that writes that day as its own token
|
|
275
|
+
(`{date+1}`) is recorded as one token. It was recorded as
|
|
276
|
+
"Thursday, {date+1:MMMM D, YYYY}", which the next day's page no
|
|
277
|
+
longer shows.
|
|
278
|
+
- `saffron run --plan` counts stability proofs among the proof replays.
|
|
279
|
+
- A drag the repair no longer makes is a behavior change only, not also
|
|
280
|
+
a "requirement not kept" note.
|
|
281
|
+
- **After a review of the unreleased work.**
|
|
282
|
+
- `verify` says when the run produced no results (`runProblem`, and an
|
|
283
|
+
MCP error), instead of answering with an empty pass, and reads only
|
|
284
|
+
the rows its own run wrote.
|
|
285
|
+
- A recording deleted outright counts as losing every check it made, in
|
|
286
|
+
`saffron coverage --base` and in `verify`.
|
|
287
|
+
- Every copy of a request sent at once has its credentials masked in
|
|
288
|
+
the transcript, not only the copy the step's checks read.
|
|
289
|
+
- `maxCostUsd` warns that Codex, Antigravity and Cursor report no price,
|
|
290
|
+
so it cannot stop their sessions.
|
|
291
|
+
- A transport module's `browser()` and `api()` hooks are asked once per
|
|
292
|
+
run, however many workers start.
|
|
293
|
+
- The command a run records hides a URL's password and writes an
|
|
294
|
+
environment variable's value as its `{env:NAME}`.
|
|
295
|
+
- `reset` runs before a recording only when an agent session can follow,
|
|
296
|
+
including when the agent's provider cannot start (a missing SDK).
|
|
297
|
+
- `verify` hides every environment value in the run's output it quotes,
|
|
298
|
+
so a secret a failed setup printed never reaches the agent.
|
|
299
|
+
- The recorded command masks only secret-like variables (a path under
|
|
300
|
+
`HOME` stays as typed) and keeps a glob quoted.
|
|
301
|
+
- A transport hook that failed is asked again by the next caller.
|
|
302
|
+
- The `maxCostUsd` warning for an unpriced provider is said once, at its
|
|
303
|
+
first agent session, not in every run.
|
|
304
|
+
- **A pluggable transport for isolated environments.** `transport` in
|
|
305
|
+
saffron.config.json connects to a browser already running elsewhere (a
|
|
306
|
+
Chromium by `cdpEndpoint`, which records and replays, or a Playwright
|
|
307
|
+
server by `wsEndpoint`, which replays), sends API requests through a
|
|
308
|
+
proxy with extra headers and self-signed certificates accepted, and
|
|
309
|
+
takes a project module whose `browser()` and `api()` hooks decide both
|
|
310
|
+
at run time, so a Docker setup needs no custom integration.
|
|
311
|
+
- **Verification for coding agents over MCP.** The new `verify` tool
|
|
312
|
+
replays the scenarios an agent names, or the whole suite, at zero AI
|
|
313
|
+
(`saffron run --no-agent`), and answers in a versioned shape: the commit
|
|
314
|
+
and environment, each check's verdict and evidence, what could not be
|
|
315
|
+
verified, proposals waiting for a person, and the spec's changes against
|
|
316
|
+
the reviewed coverage map committed at a git ref, so a weakened or
|
|
317
|
+
removed check shows even when everything is green.
|
|
318
|
+
- **Requests sent at once, and prerequisites an environment lacks.** An
|
|
319
|
+
API step can send a request several times at once (`concurrent`, with
|
|
320
|
+
the `statuses` the copies answer in any order), so a double submission
|
|
321
|
+
that books twice is a red; the agent records it for steps such as "I
|
|
322
|
+
submit the booking twice at once". Such recordings are version 12. A
|
|
323
|
+
scenario tagged `@requires:<name>` is not run where `capabilities` in
|
|
324
|
+
saffron.config.json sets that name to false: it is red, blocked by the
|
|
325
|
+
environment, naming the missing prerequisite, instead of being healed
|
|
326
|
+
around it.
|
|
327
|
+
- **A run report built for triage.** The HTML report lists every
|
|
328
|
+
scenario in one scrollable list, failures first, then proposals, then
|
|
329
|
+
passes, with a search box and a filter per status; the scenario chosen
|
|
330
|
+
shows beside it. Failures with the same error, and proposals with the
|
|
331
|
+
same change, are linked to each other, with one command for them all
|
|
332
|
+
(`saffron run` for the failures, `saffron accept` with their files for
|
|
333
|
+
the proposals). A run map below shows one cell per scenario by feature
|
|
334
|
+
file, coloured by status, AI cost or duration, so a run of 200
|
|
335
|
+
scenarios stays one page; a cell opens its scenario in an inspector
|
|
336
|
+
under the map (its steps as cards, the failure or the change, and who
|
|
337
|
+
shares it), leaving the list's choice as it was. Without scripts the page still reads: every
|
|
338
|
+
scenario is shown in turn. Under the map, widgets show setup and
|
|
339
|
+
teardown (a failed command first and flagged in the header, the ones
|
|
340
|
+
that went well folded), the workers' timing (saying when they did very
|
|
341
|
+
different amounts of work) and the pass rate of the last runs, the
|
|
342
|
+
newest marked. A duration over a minute reads as minutes and seconds,
|
|
343
|
+
and scenarios sharing a cause get a button for the command that deals
|
|
344
|
+
with them all, the command itself still on the page.
|
|
345
|
+
- **A compatibility page per release.** [Compatibility](COMPATIBILITY.md)
|
|
346
|
+
lists the Node.js, operating systems, Playwright and browsers, each
|
|
347
|
+
agent's SDK (the range accepted and the version tested) and the recording
|
|
348
|
+
versions this release reads and writes, generated from the release's own
|
|
349
|
+
code and lockfile. The agent skill names the saffron-ai version it
|
|
350
|
+
describes.
|
|
351
|
+
- **A recording-quality benchmark.** `npm run benchmark` replays the
|
|
352
|
+
example suite's recordings several times on the demo app (flaky
|
|
353
|
+
scenarios), and against deliberately broken versions of it (false
|
|
354
|
+
greens); `--record` also records the suite afresh with the agent and
|
|
355
|
+
measures the share verified on the first attempt, the share held for
|
|
356
|
+
review, and the time and cost. The results are published in the docs
|
|
357
|
+
([benchmark](BENCHMARK.md)): today 66 of 66 replays green and no false
|
|
358
|
+
green on 8 broken apps.
|
|
359
|
+
- **A coverage map per pull request.** `saffron coverage` maps each
|
|
360
|
+
requirement ID (a tag matching `requirementTag`, such as `@REQ-12`) to
|
|
361
|
+
its scenarios, the checks their recordings make and the last run's
|
|
362
|
+
verdict. `--write` saves it as `.saffron/coverage.json` to commit, and
|
|
363
|
+
`--base origin/main` compares with the target branch's map: scenarios
|
|
364
|
+
added and removed, requirements no longer covered and checks a recording
|
|
365
|
+
no longer makes, with `--markdown` for the pull request and
|
|
366
|
+
`--fail-on-loss` to gate on it.
|
|
367
|
+
- **Side effects are counted, and can be reset.** `saffron run --plan`
|
|
368
|
+
marks the scenarios that change the app's data (a write request in
|
|
369
|
+
their recording, a unique value, or an `@side-effects` tag) and how many
|
|
370
|
+
times the run executes each: a recording runs once, its proof, every
|
|
371
|
+
stability proof and every sensitivity check again. The new `reset`
|
|
372
|
+
setting runs your commands before each of those fresh executions (never
|
|
373
|
+
before an ordinary replay), with `SAFFRON_SCENARIO`, `SAFFRON_FEATURE`
|
|
374
|
+
and `SAFFRON_RESET_FOR`. A failed reset stops that recording, or fails
|
|
375
|
+
that proof, and the run goes on.
|
|
376
|
+
- **Stability proofs.** `stabilityProofs` (`--stability-proofs n`) replays
|
|
377
|
+
a recording that passed its proof n more times, each in a fresh
|
|
378
|
+
execution, before it is filed. The proposal, the run summary and the
|
|
379
|
+
review page tell "passed one proof" from "passed the stability proofs",
|
|
380
|
+
and a recording that fails one gets an `unstable recording:` note, left
|
|
381
|
+
out of `accept --all` as the editor plugins mark it.
|
|
382
|
+
- **A recording plan and spending limits.** `saffron run --plan` prints
|
|
383
|
+
what the run would do and stops, with no browser or agent: what replays,
|
|
384
|
+
what records or records again (with the reason), what is refused, how
|
|
385
|
+
many agent sessions and proof replays that takes, and new steps several
|
|
386
|
+
scenarios share, to record and accept first. `--json` and the MCP tool
|
|
387
|
+
`run_plan` give the same as data. `maxCostUsd` (`--max-cost`) and
|
|
388
|
+
`maxAgentMinutes` (`--max-agent-minutes`) stop new agent sessions once a
|
|
389
|
+
run reaches them, and the summary splits AI cost into recording, repair
|
|
390
|
+
and failed attempts.
|
|
391
|
+
- **Every repair is checked against what its recording bound.** From the
|
|
392
|
+
committed recording, Saffron lists each value a step enters, each request
|
|
393
|
+
it sends and each check it makes, and finds the repaired action that
|
|
394
|
+
keeps each one. `saffron diff`, its JSON (`contract`) and the review page
|
|
395
|
+
show the list with that evidence. A repair that no longer sends a request
|
|
396
|
+
or makes a check gets a `requirement not kept:` note, and `accept --all`
|
|
397
|
+
leaves it out for review, as the editor plugins mark it.
|
|
398
|
+
- **The run summary says which run it counts.** It names the invocation
|
|
399
|
+
(`this run (\`saffron run ...\`, 12 scenarios, 10:04)`), splits pending
|
|
400
|
+
review by proof passed, failed or not run, and splits failures into the
|
|
401
|
+
app's and those the recorder, the agent's provider or the environment
|
|
402
|
+
caused (`blocked` in the report JSON). A run of named scenarios also
|
|
403
|
+
prints the combined report's counts. The report shows the command too.
|
|
404
|
+
- **A review page for proposals.** `saffron diff --html` writes
|
|
405
|
+
`.saffron/reports/review.html`: each Gherkin step beside the action it
|
|
406
|
+
records today and the one proposed, the target, what the proof captured
|
|
407
|
+
and any note about that step, so "enter an expiry date" becoming "check
|
|
408
|
+
an expiry hint" shows in the step's own row. The last run's screenshot of
|
|
409
|
+
the page before the agent took over is included. `saffron diff` and
|
|
410
|
+
`--json` show those notes under their step too.
|
|
411
|
+
- **Dependencies.** saffron-ai now needs `zod` 4.6.7 or later, so
|
|
412
|
+
upgrading it in a project with a lockfile also brings the fixed `zod`.
|
|
413
|
+
The other packages `npm audit` may report after an upgrade
|
|
414
|
+
(`body-parser`, `qs`, `hono`, `ip-address`, `fast-uri`) come through
|
|
415
|
+
Saffron's dependencies and are already allowed at their fixed versions:
|
|
416
|
+
`npm audit fix` updates them.
|
|
417
|
+
|
|
418
|
+
Editors (VS Code extension 0.7.5, JetBrains plugin 0.7.6)
|
|
419
|
+
|
|
420
|
+
- **Two more notes hold a proposal for review.** A proposal with a
|
|
421
|
+
"requirement not kept:" note (a repair that no longer sends a request or
|
|
422
|
+
makes a check its recording did) or an "unstable recording:" note (a
|
|
423
|
+
recording that failed an extra stability proof) is held from Accept
|
|
424
|
+
All, as `saffron accept --all` holds it. The new dashboard needs no
|
|
425
|
+
editor update: both show the runner's page.
|
|
426
|
+
|
|
8
427
|
## 1.2.1: Sturdier recordings, and API keys for Codex and Antigravity
|
|
9
428
|
|
|
10
429
|
Released 2026-10-28.
|