saffron-ai 1.2.0 → 1.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +377 -1
- package/README.md +4 -4
- package/dist-pkg/cli.js +833 -462
- package/package.json +25 -9
- package/skills/saffron/SKILL.md +9 -2
- package/skills/saffron/references/api.md +6 -4
- package/skills/saffron/references/config.md +7 -6
- package/templates/playwright/saffron-helpers.ts +8 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,7 +5,375 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
-
##
|
|
8
|
+
## Unreleased
|
|
9
|
+
|
|
10
|
+
## 1.2.2: Triage in the report, a new dashboard, and verification for coding agents
|
|
11
|
+
|
|
12
|
+
Released 2026-10-28.
|
|
13
|
+
|
|
14
|
+
Runner
|
|
15
|
+
|
|
16
|
+
- **Proposals are reviewed in the run report.** A scenario with a proposal
|
|
17
|
+
has a `run | review` tab pair. The review tab holds what used to sit
|
|
18
|
+
among the run's panels: every step beside what it records today and what
|
|
19
|
+
is proposed (unchanged steps dimmed), the notes on a step, the agent's
|
|
20
|
+
account, the page before the agent took over, the proof and stability
|
|
21
|
+
proofs, and the commands to accept or reject, each with a copy button.
|
|
22
|
+
The header has a "N proposals to review" chip that opens the first one.
|
|
23
|
+
The page before the agent took over is embedded once, as before.
|
|
24
|
+
`saffron diff --html` is unchanged. The report stays read only.
|
|
25
|
+
- **A new dashboard, in both editors and `saffron dashboard`.** The page
|
|
26
|
+
opens on the suite's health and a history grid: each scenario that is
|
|
27
|
+
failing, flaky or healed again and again gets one cell per run (last 20),
|
|
28
|
+
so a new break, a flaky test and a drifting page read at a glance. Click
|
|
29
|
+
a row for its error, and a scenario's name to open it in the editor.
|
|
30
|
+
A scenario that fits several chips (failing and flaky, say) is listed
|
|
31
|
+
once, under the most urgent group, and is found under each chip; a
|
|
32
|
+
group's count follows the chip, and a chip with nothing under it cannot
|
|
33
|
+
be pressed. The grid's columns are the last 20 runs a listed
|
|
34
|
+
scenario ran in, so runs from an editor's Run button of other scenarios
|
|
35
|
+
no longer stretch it with empty columns.
|
|
36
|
+
Slowest, tags and coverage gaps sit beside it. The pass-rate and
|
|
37
|
+
execution-time bar charts are gone. The editors need no update: they show
|
|
38
|
+
the runner's page.
|
|
39
|
+
- **Found in a wider test pass.**
|
|
40
|
+
- A day the page spells with its weekday ("Thursday, October 29,
|
|
41
|
+
2026") in a scenario that writes that day as its own token
|
|
42
|
+
(`{date+1}`) is recorded as one token. It was recorded as
|
|
43
|
+
"Thursday, {date+1:MMMM D, YYYY}", which the next day's page no
|
|
44
|
+
longer shows.
|
|
45
|
+
- `saffron run --plan` counts stability proofs among the proof replays.
|
|
46
|
+
- A drag the repair no longer makes is a behavior change only, not also
|
|
47
|
+
a "requirement not kept" note.
|
|
48
|
+
- **After a review of the unreleased work.**
|
|
49
|
+
- `verify` says when the run produced no results (`runProblem`, and an
|
|
50
|
+
MCP error), instead of answering with an empty pass, and reads only
|
|
51
|
+
the rows its own run wrote.
|
|
52
|
+
- A recording deleted outright counts as losing every check it made, in
|
|
53
|
+
`saffron coverage --base` and in `verify`.
|
|
54
|
+
- Every copy of a request sent at once has its credentials masked in
|
|
55
|
+
the transcript, not only the copy the step's checks read.
|
|
56
|
+
- `maxCostUsd` warns that Codex, Antigravity and Cursor report no price,
|
|
57
|
+
so it cannot stop their sessions.
|
|
58
|
+
- A transport module's `browser()` and `api()` hooks are asked once per
|
|
59
|
+
run, however many workers start.
|
|
60
|
+
- The command a run records hides a URL's password and writes an
|
|
61
|
+
environment variable's value as its `{env:NAME}`.
|
|
62
|
+
- `reset` runs before a recording only when an agent session can follow,
|
|
63
|
+
including when the agent's provider cannot start (a missing SDK).
|
|
64
|
+
- `verify` hides every environment value in the run's output it quotes,
|
|
65
|
+
so a secret a failed setup printed never reaches the agent.
|
|
66
|
+
- The recorded command masks only secret-like variables (a path under
|
|
67
|
+
`HOME` stays as typed) and keeps a glob quoted.
|
|
68
|
+
- A transport hook that failed is asked again by the next caller.
|
|
69
|
+
- The `maxCostUsd` warning for an unpriced provider is said once, at its
|
|
70
|
+
first agent session, not in every run.
|
|
71
|
+
- **A pluggable transport for isolated environments.** `transport` in
|
|
72
|
+
saffron.config.json connects to a browser already running elsewhere (a
|
|
73
|
+
Chromium by `cdpEndpoint`, which records and replays, or a Playwright
|
|
74
|
+
server by `wsEndpoint`, which replays), sends API requests through a
|
|
75
|
+
proxy with extra headers and self-signed certificates accepted, and
|
|
76
|
+
takes a project module whose `browser()` and `api()` hooks decide both
|
|
77
|
+
at run time, so a Docker setup needs no custom integration.
|
|
78
|
+
- **Verification for coding agents over MCP.** The new `verify` tool
|
|
79
|
+
replays the scenarios an agent names, or the whole suite, at zero AI
|
|
80
|
+
(`saffron run --no-agent`), and answers in a versioned shape: the commit
|
|
81
|
+
and environment, each check's verdict and evidence, what could not be
|
|
82
|
+
verified, proposals waiting for a person, and the spec's changes against
|
|
83
|
+
the reviewed coverage map committed at a git ref, so a weakened or
|
|
84
|
+
removed check shows even when everything is green.
|
|
85
|
+
- **Requests sent at once, and prerequisites an environment lacks.** An
|
|
86
|
+
API step can send a request several times at once (`concurrent`, with
|
|
87
|
+
the `statuses` the copies answer in any order), so a double submission
|
|
88
|
+
that books twice is a red; the agent records it for steps such as "I
|
|
89
|
+
submit the booking twice at once". Such recordings are version 12. A
|
|
90
|
+
scenario tagged `@requires:<name>` is not run where `capabilities` in
|
|
91
|
+
saffron.config.json sets that name to false: it is red, blocked by the
|
|
92
|
+
environment, naming the missing prerequisite, instead of being healed
|
|
93
|
+
around it.
|
|
94
|
+
- **A run report built for triage.** The HTML report lists every
|
|
95
|
+
scenario in one scrollable list, failures first, then proposals, then
|
|
96
|
+
passes, with a search box and a filter per status; the scenario chosen
|
|
97
|
+
shows beside it. Failures with the same error, and proposals with the
|
|
98
|
+
same change, are linked to each other, with one command for them all
|
|
99
|
+
(`saffron run` for the failures, `saffron accept` with their files for
|
|
100
|
+
the proposals). A run map below shows one cell per scenario by feature
|
|
101
|
+
file, coloured by status, AI cost or duration, so a run of 200
|
|
102
|
+
scenarios stays one page; a cell opens its scenario in an inspector
|
|
103
|
+
under the map (its steps as cards, the failure or the change, and who
|
|
104
|
+
shares it), leaving the list's choice as it was. Without scripts the page still reads: every
|
|
105
|
+
scenario is shown in turn. Under the map, widgets show setup and
|
|
106
|
+
teardown (a failed command first and flagged in the header, the ones
|
|
107
|
+
that went well folded), the workers' timing (saying when they did very
|
|
108
|
+
different amounts of work) and the pass rate of the last runs, the
|
|
109
|
+
newest marked. A duration over a minute reads as minutes and seconds,
|
|
110
|
+
and scenarios sharing a cause get a button for the command that deals
|
|
111
|
+
with them all, the command itself still on the page.
|
|
112
|
+
- **A compatibility page per release.** [Compatibility](COMPATIBILITY.md)
|
|
113
|
+
lists the Node.js, operating systems, Playwright and browsers, each
|
|
114
|
+
agent's SDK (the range accepted and the version tested) and the recording
|
|
115
|
+
versions this release reads and writes, generated from the release's own
|
|
116
|
+
code and lockfile. The agent skill names the saffron-ai version it
|
|
117
|
+
describes.
|
|
118
|
+
- **A recording-quality benchmark.** `npm run benchmark` replays the
|
|
119
|
+
example suite's recordings several times on the demo app (flaky
|
|
120
|
+
scenarios), and against deliberately broken versions of it (false
|
|
121
|
+
greens); `--record` also records the suite afresh with the agent and
|
|
122
|
+
measures the share verified on the first attempt, the share held for
|
|
123
|
+
review, and the time and cost. The results are published in the docs
|
|
124
|
+
([benchmark](BENCHMARK.md)): today 66 of 66 replays green and no false
|
|
125
|
+
green on 8 broken apps.
|
|
126
|
+
- **A coverage map per pull request.** `saffron coverage` maps each
|
|
127
|
+
requirement ID (a tag matching `requirementTag`, such as `@REQ-12`) to
|
|
128
|
+
its scenarios, the checks their recordings make and the last run's
|
|
129
|
+
verdict. `--write` saves it as `.saffron/coverage.json` to commit, and
|
|
130
|
+
`--base origin/main` compares with the target branch's map: scenarios
|
|
131
|
+
added and removed, requirements no longer covered and checks a recording
|
|
132
|
+
no longer makes, with `--markdown` for the pull request and
|
|
133
|
+
`--fail-on-loss` to gate on it.
|
|
134
|
+
- **Side effects are counted, and can be reset.** `saffron run --plan`
|
|
135
|
+
marks the scenarios that change the app's data (a write request in
|
|
136
|
+
their recording, a unique value, or an `@side-effects` tag) and how many
|
|
137
|
+
times the run executes each: a recording runs once, its proof, every
|
|
138
|
+
stability proof and every sensitivity check again. The new `reset`
|
|
139
|
+
setting runs your commands before each of those fresh executions (never
|
|
140
|
+
before an ordinary replay), with `SAFFRON_SCENARIO`, `SAFFRON_FEATURE`
|
|
141
|
+
and `SAFFRON_RESET_FOR`. A failed reset stops that recording, or fails
|
|
142
|
+
that proof, and the run goes on.
|
|
143
|
+
- **Stability proofs.** `stabilityProofs` (`--stability-proofs n`) replays
|
|
144
|
+
a recording that passed its proof n more times, each in a fresh
|
|
145
|
+
execution, before it is filed. The proposal, the run summary and the
|
|
146
|
+
review page tell "passed one proof" from "passed the stability proofs",
|
|
147
|
+
and a recording that fails one gets an `unstable recording:` note, left
|
|
148
|
+
out of `accept --all` as the editor plugins mark it.
|
|
149
|
+
- **A recording plan and spending limits.** `saffron run --plan` prints
|
|
150
|
+
what the run would do and stops, with no browser or agent: what replays,
|
|
151
|
+
what records or records again (with the reason), what is refused, how
|
|
152
|
+
many agent sessions and proof replays that takes, and new steps several
|
|
153
|
+
scenarios share, to record and accept first. `--json` and the MCP tool
|
|
154
|
+
`run_plan` give the same as data. `maxCostUsd` (`--max-cost`) and
|
|
155
|
+
`maxAgentMinutes` (`--max-agent-minutes`) stop new agent sessions once a
|
|
156
|
+
run reaches them, and the summary splits AI cost into recording, repair
|
|
157
|
+
and failed attempts.
|
|
158
|
+
- **Every repair is checked against what its recording bound.** From the
|
|
159
|
+
committed recording, Saffron lists each value a step enters, each request
|
|
160
|
+
it sends and each check it makes, and finds the repaired action that
|
|
161
|
+
keeps each one. `saffron diff`, its JSON (`contract`) and the review page
|
|
162
|
+
show the list with that evidence. A repair that no longer sends a request
|
|
163
|
+
or makes a check gets a `requirement not kept:` note, and `accept --all`
|
|
164
|
+
leaves it out for review, as the editor plugins mark it.
|
|
165
|
+
- **The run summary says which run it counts.** It names the invocation
|
|
166
|
+
(`this run (\`saffron run ...\`, 12 scenarios, 10:04)`), splits pending
|
|
167
|
+
review by proof passed, failed or not run, and splits failures into the
|
|
168
|
+
app's and those the recorder, the agent's provider or the environment
|
|
169
|
+
caused (`blocked` in the report JSON). A run of named scenarios also
|
|
170
|
+
prints the combined report's counts. The report shows the command too.
|
|
171
|
+
- **A review page for proposals.** `saffron diff --html` writes
|
|
172
|
+
`.saffron/reports/review.html`: each Gherkin step beside the action it
|
|
173
|
+
records today and the one proposed, the target, what the proof captured
|
|
174
|
+
and any note about that step, so "enter an expiry date" becoming "check
|
|
175
|
+
an expiry hint" shows in the step's own row. The last run's screenshot of
|
|
176
|
+
the page before the agent took over is included. `saffron diff` and
|
|
177
|
+
`--json` show those notes under their step too.
|
|
178
|
+
- **Dependencies.** saffron-ai now needs `zod` 4.6.7 or later, so
|
|
179
|
+
upgrading it in a project with a lockfile also brings the fixed `zod`.
|
|
180
|
+
The other packages `npm audit` may report after an upgrade
|
|
181
|
+
(`body-parser`, `qs`, `hono`, `ip-address`, `fast-uri`) come through
|
|
182
|
+
Saffron's dependencies and are already allowed at their fixed versions:
|
|
183
|
+
`npm audit fix` updates them.
|
|
184
|
+
|
|
185
|
+
Editors (VS Code extension 0.7.5, JetBrains plugin 0.7.6)
|
|
186
|
+
|
|
187
|
+
- **Two more notes hold a proposal for review.** A proposal with a
|
|
188
|
+
"requirement not kept:" note (a repair that no longer sends a request or
|
|
189
|
+
makes a check its recording did) or an "unstable recording:" note (a
|
|
190
|
+
recording that failed an extra stability proof) is held from Accept
|
|
191
|
+
All, as `saffron accept --all` holds it. The new dashboard needs no
|
|
192
|
+
editor update: both show the runner's page.
|
|
193
|
+
|
|
194
|
+
## 1.2.1: Sturdier recordings, and API keys for Codex and Antigravity
|
|
195
|
+
|
|
196
|
+
Released 2026-10-28.
|
|
197
|
+
|
|
198
|
+
Runner
|
|
199
|
+
|
|
200
|
+
- **After a review of the unreleased work.**
|
|
201
|
+
- An Outline row's date that the page spells out (`Friday, October 30,
|
|
202
|
+
2026`) is recorded as that row's own date token, so another row never
|
|
203
|
+
replays on it and the same row follows the day without recording
|
|
204
|
+
again. A day the page works out from it (a checkout two days on)
|
|
205
|
+
follows the row too. A recording that kept the date as text records
|
|
206
|
+
once more. Such a recording is schema version 11, which 1.2.0 refuses.
|
|
207
|
+
Another row with a different date there fails with its reason; it
|
|
208
|
+
needs a Scenario of its own. A day shown in a step that does not use
|
|
209
|
+
the column (today's date in a banner) no longer ties the Outline to
|
|
210
|
+
one row.
|
|
211
|
+
- A date the scenario writes itself (`2026-10-30`) stays that date when
|
|
212
|
+
the page spells it otherwise ("Friday, October 30, 2026"), instead of
|
|
213
|
+
moving with the calendar from the next day on.
|
|
214
|
+
- A token used as a key in a whole-JSON Examples column is refused
|
|
215
|
+
instead of being sent as a broken field name.
|
|
216
|
+
- `saffron author` makes the checks a recording makes before it starts
|
|
217
|
+
the agent, except agy's rules for Saffron's tools, which it does not
|
|
218
|
+
use: with `antigravityApiKey` set and no key it refuses instead
|
|
219
|
+
of signing in with the Google login, and a failed session prints its
|
|
220
|
+
reason instead of a stack trace.
|
|
221
|
+
- A token outside any string in a whole-JSON Examples column
|
|
222
|
+
(`{"qty": {captured:count}}`) is a value of its own, so a secret
|
|
223
|
+
beside it can no longer add a field. A cell that is neither JSON nor
|
|
224
|
+
one whole token is refused instead of being filled in first. The same
|
|
225
|
+
holds in exported tests.
|
|
226
|
+
- Captured values in a proposal mask a credential an API handed out.
|
|
227
|
+
- A secret inside a whole-JSON Examples column can no longer rewrite
|
|
228
|
+
the payload.
|
|
229
|
+
- "Should differ" compares references and counts (`ABC-123`,
|
|
230
|
+
`1 night`) as text, refusing only amounts in different currencies or
|
|
231
|
+
a price against a label, and a brief bad reading no longer ends the
|
|
232
|
+
check.
|
|
233
|
+
- A heal that swaps a tick, untick, upload or drag for another input is
|
|
234
|
+
flagged as a behavior change.
|
|
235
|
+
- The sensitivity check gives each check a second, skips a `Then` with
|
|
236
|
+
no check or in a session not yet open, and cannot abort a run. The
|
|
237
|
+
report counts only the proposals it checked.
|
|
238
|
+
- `expectDisabled` is decided by the recorded element, not a fallback.
|
|
239
|
+
- Slashed dates are read in the locale's order, a UK date with its
|
|
240
|
+
weekday becomes a token, and a month the scenario names is not a
|
|
241
|
+
volatile locator.
|
|
242
|
+
- **The npm page says what Saffron is.** The package description no longer
|
|
243
|
+
calls it an AI-fallback test runner, its keywords cover the terms people
|
|
244
|
+
search for, it names its author, and the README names Antigravity among
|
|
245
|
+
the agents.
|
|
246
|
+
- **`prune --json` no longer calls a broken file's recordings orphans.**
|
|
247
|
+
While a feature file fails to parse, prune already deleted nothing, but
|
|
248
|
+
its JSON listed the hidden scenarios' live recordings as orphans. Now
|
|
249
|
+
they are in `orphanScan.ownershipUnknown`, beside the parse `errors`;
|
|
250
|
+
other files' orphans stay in `orphans`, and the JSON has the same shape
|
|
251
|
+
on a healthy project. `saffron status --json` splits them the same way,
|
|
252
|
+
where it listed no orphans at all.
|
|
253
|
+
- **Volatile locators are caught while recording.** A date the agent
|
|
254
|
+
picked that the app spells out (`October 30, 2026`, `30.10.2026`) is
|
|
255
|
+
recorded as a `{date}` token, as ISO dates already were, and a day the
|
|
256
|
+
scenario's own token wrote keeps that token. A generated id in a
|
|
257
|
+
selector, frame or test id, a UUID in a name, or a month heading is
|
|
258
|
+
refused once at `saffron_done` so the agent finds a stable target; what
|
|
259
|
+
is left gets a `volatile locator:` note, and `accept --all` leaves that
|
|
260
|
+
proposal to be accepted by file.
|
|
261
|
+
- **An Examples column can hold a whole JSON payload.** A request whose
|
|
262
|
+
body, or a part of it, is the JSON a column the step names holds is
|
|
263
|
+
recorded as that column, so each row sends its own payload. Before, the
|
|
264
|
+
payload was kept as fixed fields and every other row was refused. The
|
|
265
|
+
proposal names the body each row sends, and a cell that is not JSON fails
|
|
266
|
+
its row by name. Such a recording is schema version 10.
|
|
267
|
+
- **Comparisons compare like with like.** A "should differ" check between
|
|
268
|
+
two amounts compares the numbers, so the same price drawn in another
|
|
269
|
+
format has not changed. Two values of different kinds fail at once and
|
|
270
|
+
say why: `SEK 1,295` against `NOK 1,347` differ by currency alone, and a
|
|
271
|
+
whole button label against a price by its label. `saffron diff` and the
|
|
272
|
+
report show what each capture read in the proof replay beside its step.
|
|
273
|
+
A step that says an element is selectable, available or disabled records
|
|
274
|
+
an enabled or disabled check (`expectEnabled`, `expectDisabled`), where it
|
|
275
|
+
recorded visibility only; such a recording is schema version 9.
|
|
276
|
+
- **Recorded checks must be able to fail.** A proof shows a recording
|
|
277
|
+
replays, not that its checks notice anything. After a proof passes,
|
|
278
|
+
Saffron replays the scenario once more with no AI and runs each `Then`
|
|
279
|
+
on the page as it was just before the action step it follows. A check
|
|
280
|
+
that passes there too gets an `insensitive check:` note (picking the day
|
|
281
|
+
that was already picked, a price compared with a value it always had),
|
|
282
|
+
and `accept --all` leaves that proposal to be accepted by file. The
|
|
283
|
+
report shows "checks proven sensitive in N of M" beside the proofs that
|
|
284
|
+
passed.
|
|
285
|
+
- **Heals keep what the scenario tests.** A heal serves the scenario's
|
|
286
|
+
intent first, then the step's, and the step's wording last: it may change
|
|
287
|
+
how a step is done, never what the scenario exercises. The agent is told
|
|
288
|
+
never to replace typed data with a check of a hint, or a real payment or
|
|
289
|
+
sign-in route with a simulator's button, even when the final `Then`
|
|
290
|
+
passes, and to fail and name a missing prerequisite instead of working
|
|
291
|
+
around it. Saffron checks this itself: a healed step that no longer
|
|
292
|
+
enters, picks or uploads what the committed recording did gets a
|
|
293
|
+
`behavior change:` note naming what it checks instead, and
|
|
294
|
+
`accept --all` leaves that proposal to be accepted by file. A value moved
|
|
295
|
+
to another control (a text field that became a dropdown) is still
|
|
296
|
+
entered and raises no note.
|
|
297
|
+
- **iframes are recorded with their whole scope.** An element in an iframe
|
|
298
|
+
inside another iframe keeps both, and one of several alike iframes
|
|
299
|
+
(previews, cards) keeps its position, so replay acts in the iframe the
|
|
300
|
+
agent used. Before, an iframe picked with `first()` or `nth()` lost its
|
|
301
|
+
position and replay could act in another one, and an element in a nested
|
|
302
|
+
iframe recorded no action at all. Code the recorder cannot read, such as
|
|
303
|
+
an element reached through a variable or an iframe picked by a filter, now
|
|
304
|
+
fails the recording and names the interaction instead of being left out.
|
|
305
|
+
`saffron_assert` takes the same frame forms. A recording with a nested or
|
|
306
|
+
positioned iframe is written as schema version 8, which saffron-ai 1.2.0
|
|
307
|
+
and earlier ask to be updated for; others keep their version.
|
|
308
|
+
- **Codex and Antigravity on an API key.** A machine with no ChatGPT or
|
|
309
|
+
Google login, a CI job say, records and heals with Codex on
|
|
310
|
+
`CODEX_API_KEY` (or `OPENAI_API_KEY`) and with Antigravity on
|
|
311
|
+
`GEMINI_API_KEY` (or `GOOGLE_API_KEY`), billed to that API account. A
|
|
312
|
+
login on the machine wins over a key, so a key set for other tools never
|
|
313
|
+
moves your runs off your plan. Whether there is a login is asked only
|
|
314
|
+
when the first session starts, and only when a key is set, without
|
|
315
|
+
holding up other sessions; only the vendor saying you are not signed in
|
|
316
|
+
counts, so a timeout or a network error keeps the login. The run's first
|
|
317
|
+
line says how sessions will sign in, and the first session says which one
|
|
318
|
+
it used ("Codex: signed in with API key (CODEX_API_KEY)"); a key stored by
|
|
319
|
+
`codex login --with-api-key` reads as one, not as a ChatGPT login.
|
|
320
|
+
`saffron login codex` and `saffron login antigravity` say which sign-in
|
|
321
|
+
sessions will use. An expired login still counts as a login: the session
|
|
322
|
+
it fails says the login won over the key, and how to sign in again or use
|
|
323
|
+
the key. Codex's `forced_login_method` decides alone: a key never counts
|
|
324
|
+
as a sign-in under `"chatgpt"`, `"api"` with no key in the environment
|
|
325
|
+
or stored by `codex login --with-api-key` (a stored ChatGPT login does
|
|
326
|
+
not count) is reported as a missing key, and a refused session names the setting. Antigravity's
|
|
327
|
+
equivalent is `"antigravityApiKey": true` in `saffron.config.json`: the
|
|
328
|
+
key is used even beside an agy login, by every command, and a missing
|
|
329
|
+
key is reported and starts no session. A key reaches the
|
|
330
|
+
vendor alone: an error that quotes it, whole or cut short, shows its
|
|
331
|
+
variable's name, and it is never written into a recording or report.
|
|
332
|
+
Codex on a key is verified live; Antigravity on a key is not yet.
|
|
333
|
+
- **A numeric credential is never kept in a recording.** A login's token
|
|
334
|
+
that is only digits (a one-time code, a PIN) is written as its
|
|
335
|
+
`{captured:name}` in a form field, a multipart part or a JSON number
|
|
336
|
+
wherever it stands, whatever the field is called. Before, a field whose
|
|
337
|
+
name did not name an id kept the value.
|
|
338
|
+
- **An API is named by the words that call it.** The check that the APIs a
|
|
339
|
+
scenario needs are up reads a named API from its recording, or from steps
|
|
340
|
+
that call it one ("the orders API", "the orders service"): "I list orders
|
|
341
|
+
through the default API" no longer waits on a service named `orders`.
|
|
342
|
+
- **A changed contract the API breaks is red.** When a recording is checked
|
|
343
|
+
again against a changed API contract and the API no longer answers as the
|
|
344
|
+
new contract says, the scenario is red and the run exits 1, as before
|
|
345
|
+
1.2.0, with no proposal filed (nothing was recorded again). In 1.2.0 it
|
|
346
|
+
was filed as an unverified proposal and the run passed. One that passes is
|
|
347
|
+
filed for review as before, and its run line says it replayed from the
|
|
348
|
+
recording, not that an agent recorded it. `saffron status` says such a
|
|
349
|
+
recording is checked again on the next run instead of "will re-record",
|
|
350
|
+
and the Saffron skill describes it.
|
|
351
|
+
- **A malformed `{unique:...}` stops the run before it starts.** An
|
|
352
|
+
unclosed token, an empty or invalid name, or one an Examples row fills in
|
|
353
|
+
with an empty cell (`{unique:<id>}`) is named and the run exits 2, as a
|
|
354
|
+
bad `{data:...}` does, instead of costing an agent session per row.
|
|
355
|
+
- **Failure headings name the row's values.** `Check "the customer is on
|
|
356
|
+
the "Gold" plan" failed`, as the detail below it already did.
|
|
357
|
+
- **A form field that is only a number** is written as a captured value's
|
|
358
|
+
token only where its name names an id or the capture, as in a JSON body:
|
|
359
|
+
`quantity=100` beside order 100 stays 100.
|
|
360
|
+
- **A status code with its own reason** names that status whatever
|
|
361
|
+
follows: "expect 409 Conflict and a retry hint".
|
|
362
|
+
|
|
363
|
+
Editors (VS Code extension 0.7.4, JetBrains plugin 0.7.5)
|
|
364
|
+
|
|
365
|
+
- **Proposals Accept All skips are marked.** A proposal with a "behavior
|
|
366
|
+
change:", "insensitive check:" or "volatile locator:" note, or one
|
|
367
|
+
imported from a CI run or not verified, is marked as held (VS Code:
|
|
368
|
+
"held for review"; JetBrains: "held from Accept All").
|
|
369
|
+
- **Orphans while a file fails to parse.** The Orphans view lists the
|
|
370
|
+
other files' orphans and groups the broken file's recordings under
|
|
371
|
+
"Ownership unknown until it parses"; prune waits until it parses. Needs
|
|
372
|
+
saffron-ai 1.2.1.
|
|
373
|
+
- **No more endless Loading.** A status or diff that never answers stops
|
|
374
|
+
after 60 seconds with its error, and later refreshes run again.
|
|
375
|
+
|
|
376
|
+
## 1.2.0: API test automation (beta)
|
|
9
377
|
|
|
10
378
|
Released 2026-10-26.
|
|
11
379
|
|
|
@@ -171,6 +539,14 @@ Runner
|
|
|
171
539
|
proposals, proofs, heals and the verdict) and drives the browser and the
|
|
172
540
|
HTTP medium through one contract. Nothing changes for a project.
|
|
173
541
|
|
|
542
|
+
Editors (VS Code extension 0.7.3, JetBrains plugin 0.7.4)
|
|
543
|
+
|
|
544
|
+
- **An API scenario's requests in Last Run.** Last Run never opens an API
|
|
545
|
+
scenario's request transcript as a screenshot: only failure and
|
|
546
|
+
before-heal pictures open as one, as the runner picks them. Such a row
|
|
547
|
+
offers Open Replay, which steps through the requests it sent. Needs
|
|
548
|
+
saffron-ai 1.2.0.
|
|
549
|
+
|
|
174
550
|
## 1.1.0: Secrets the AI agent never sees
|
|
175
551
|
|
|
176
552
|
Released 2026-10-22.
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
Free to use · [npm: saffron-ai](https://www.npmjs.com/package/saffron-ai) · [site](https://saffron-ai.io)
|
|
6
6
|
|
|
7
|
-
Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude by default, or Codex,
|
|
7
|
+
Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude by default, or Codex, Antigravity or Cursor on the subscription you already have, driving Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**: a human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
|
|
8
8
|
|
|
9
9
|
When the UI changes and a cached step fails at runtime, the agent takes over mid-execution with full scenario context, adapts, and finishes the run, then files a **cache proposal** you accept or reject, like a snapshot test update. Reports state exactly what was adapted and what it cost.
|
|
10
10
|
|
|
@@ -17,7 +17,7 @@ When the UI changes and a cached step fails at runtime, the agent takes over mid
|
|
|
17
17
|
- **Verified proposals.** Every recording/heal is proof-replayed zero-AI before it's filed (with one bounded refinement pass on failure), so proposals arrive stamped `verified ✓` or honestly `UNVERIFIED ✗`.
|
|
18
18
|
- **Honest cost reporting.** Reports break out prompt-cache reads/writes (the real bill of agent sessions) next to the in+out token count, per scenario and in totals, and name the models used. On an API key the dollars are what was billed; on a Claude subscription the headline is the 5-hour plan window before and after the run, with the dollars as the API-equivalent.
|
|
19
19
|
- **Your subscription, your choice of agent.** Recording and healing run on the plan already logged in on the machine: Claude (default), Codex on a ChatGPT plan, Google AI Pro/Ultra through the Antigravity CLI (experimental), or Cursor. Set `"provider"` in the config; each one runs through its vendor's official client, and every provider records through the same recorder, so the caches are identical in kind.
|
|
20
|
-
- **Trend memory.** Every run appends to `.saffron/history.jsonl`; reports show deltas vs the previous run and 20-run sparklines, and Saffron flags **chronic scenarios** (healing repeatedly
|
|
20
|
+
- **Trend memory.** Every run appends to `.saffron/history.jsonl`; reports show deltas vs the previous run and 20-run sparklines, and Saffron flags **chronic scenarios** (healing repeatedly: re-record instead of paying again) plus recurring failure themes.
|
|
21
21
|
|
|
22
22
|
## Step reuse: new scenarios get cheaper as your suite grows
|
|
23
23
|
|
|
@@ -128,8 +128,8 @@ Configuration lives in `saffron.config.json` at your project root:
|
|
|
128
128
|
```
|
|
129
129
|
|
|
130
130
|
`provider` (also `--provider`) picks whose agent records and heals:
|
|
131
|
-
`claude` (default), `codex` (ChatGPT plan
|
|
132
|
-
`antigravity` (Google AI Pro/Ultra through Google's `agy` CLI,
|
|
131
|
+
`claude` (default), `codex` (ChatGPT plan or `CODEX_API_KEY`; `npm i -D "@openai/codex-sdk@>=0.158.0"`),
|
|
132
|
+
`antigravity` (Google AI Pro/Ultra or `GEMINI_API_KEY`, through Google's `agy` CLI,
|
|
133
133
|
experimental; `npx saffron login antigravity` once) or `cursor`
|
|
134
134
|
(`npm i -D "@cursor/sdk@^1.0.32"`, then `npx saffron login cursor`). `model` takes
|
|
135
135
|
that provider's model names. `npx saffron login` says what a provider
|