saffron-ai 1.2.1 → 1.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,192 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
5
5
  JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
6
6
  release heading names its main feature: `## <version>: <title>`.
7
7
 
8
+ ## Unreleased
9
+
10
+ ## 1.2.2: Triage in the report, a new dashboard, and verification for coding agents
11
+
12
+ Released 2026-10-28.
13
+
14
+ Runner
15
+
16
+ - **Proposals are reviewed in the run report.** A scenario with a proposal
17
+ has a `run | review` tab pair. The review tab holds what used to sit
18
+ among the run's panels: every step beside what it records today and what
19
+ is proposed (unchanged steps dimmed), the notes on a step, the agent's
20
+ account, the page before the agent took over, the proof and stability
21
+ proofs, and the commands to accept or reject, each with a copy button.
22
+ The header has a "N proposals to review" chip that opens the first one.
23
+ The page before the agent took over is embedded once, as before.
24
+ `saffron diff --html` is unchanged. The report stays read only.
25
+ - **A new dashboard, in both editors and `saffron dashboard`.** The page
26
+ opens on the suite's health and a history grid: each scenario that is
27
+ failing, flaky or healed again and again gets one cell per run (last 20),
28
+ so a new break, a flaky test and a drifting page read at a glance. Click
29
+ a row for its error, and a scenario's name to open it in the editor.
30
+ A scenario that fits several chips (failing and flaky, say) is listed
31
+ once, under the most urgent group, and is found under each chip; a
32
+ group's count follows the chip, and a chip with nothing under it cannot
33
+ be pressed. The grid's columns are the last 20 runs a listed
34
+ scenario ran in, so runs from an editor's Run button of other scenarios
35
+ no longer stretch it with empty columns.
36
+ Slowest, tags and coverage gaps sit beside it. The pass-rate and
37
+ execution-time bar charts are gone. The editors need no update: they show
38
+ the runner's page.
39
+ - **Found in a wider test pass.**
40
+ - A day the page spells with its weekday ("Thursday, October 29,
41
+ 2026") in a scenario that writes that day as its own token
42
+ (`{date+1}`) is recorded as one token. It was recorded as
43
+ "Thursday, {date+1:MMMM D, YYYY}", which the next day's page no
44
+ longer shows.
45
+ - `saffron run --plan` counts stability proofs among the proof replays.
46
+ - A drag the repair no longer makes is a behavior change only, not also
47
+ a "requirement not kept" note.
48
+ - **After a review of the unreleased work.**
49
+ - `verify` says when the run produced no results (`runProblem`, and an
50
+ MCP error), instead of answering with an empty pass, and reads only
51
+ the rows its own run wrote.
52
+ - A recording deleted outright counts as losing every check it made, in
53
+ `saffron coverage --base` and in `verify`.
54
+ - Every copy of a request sent at once has its credentials masked in
55
+ the transcript, not only the copy the step's checks read.
56
+ - `maxCostUsd` warns that Codex, Antigravity and Cursor report no price,
57
+ so it cannot stop their sessions.
58
+ - A transport module's `browser()` and `api()` hooks are asked once per
59
+ run, however many workers start.
60
+ - The command a run records hides a URL's password and writes an
61
+ environment variable's value as its `{env:NAME}`.
62
+ - `reset` runs before a recording only when an agent session can follow,
63
+ including when the agent's provider cannot start (a missing SDK).
64
+ - `verify` hides every environment value in the run's output it quotes,
65
+ so a secret a failed setup printed never reaches the agent.
66
+ - The recorded command masks only secret-like variables (a path under
67
+ `HOME` stays as typed) and keeps a glob quoted.
68
+ - A transport hook that failed is asked again by the next caller.
69
+ - The `maxCostUsd` warning for an unpriced provider is said once, at its
70
+ first agent session, not in every run.
71
+ - **A pluggable transport for isolated environments.** `transport` in
72
+ saffron.config.json connects to a browser already running elsewhere (a
73
+ Chromium by `cdpEndpoint`, which records and replays, or a Playwright
74
+ server by `wsEndpoint`, which replays), sends API requests through a
75
+ proxy with extra headers and self-signed certificates accepted, and
76
+ takes a project module whose `browser()` and `api()` hooks decide both
77
+ at run time, so a Docker setup needs no custom integration.
78
+ - **Verification for coding agents over MCP.** The new `verify` tool
79
+ replays the scenarios an agent names, or the whole suite, at zero AI
80
+ (`saffron run --no-agent`), and answers in a versioned shape: the commit
81
+ and environment, each check's verdict and evidence, what could not be
82
+ verified, proposals waiting for a person, and the spec's changes against
83
+ the reviewed coverage map committed at a git ref, so a weakened or
84
+ removed check shows even when everything is green.
85
+ - **Requests sent at once, and prerequisites an environment lacks.** An
86
+ API step can send a request several times at once (`concurrent`, with
87
+ the `statuses` the copies answer in any order), so a double submission
88
+ that books twice is a red; the agent records it for steps such as "I
89
+ submit the booking twice at once". Such recordings are version 12. A
90
+ scenario tagged `@requires:<name>` is not run where `capabilities` in
91
+ saffron.config.json sets that name to false: it is red, blocked by the
92
+ environment, naming the missing prerequisite, instead of being healed
93
+ around it.
94
+ - **A run report built for triage.** The HTML report lists every
95
+ scenario in one scrollable list, failures first, then proposals, then
96
+ passes, with a search box and a filter per status; the scenario chosen
97
+ shows beside it. Failures with the same error, and proposals with the
98
+ same change, are linked to each other, with one command for them all
99
+ (`saffron run` for the failures, `saffron accept` with their files for
100
+ the proposals). A run map below shows one cell per scenario by feature
101
+ file, coloured by status, AI cost or duration, so a run of 200
102
+ scenarios stays one page; a cell opens its scenario in an inspector
103
+ under the map (its steps as cards, the failure or the change, and who
104
+ shares it), leaving the list's choice as it was. Without scripts the page still reads: every
105
+ scenario is shown in turn. Under the map, widgets show setup and
106
+ teardown (a failed command first and flagged in the header, the ones
107
+ that went well folded), the workers' timing (saying when they did very
108
+ different amounts of work) and the pass rate of the last runs, the
109
+ newest marked. A duration over a minute reads as minutes and seconds,
110
+ and scenarios sharing a cause get a button for the command that deals
111
+ with them all, the command itself still on the page.
112
+ - **A compatibility page per release.** [Compatibility](COMPATIBILITY.md)
113
+ lists the Node.js, operating systems, Playwright and browsers, each
114
+ agent's SDK (the range accepted and the version tested) and the recording
115
+ versions this release reads and writes, generated from the release's own
116
+ code and lockfile. The agent skill names the saffron-ai version it
117
+ describes.
118
+ - **A recording-quality benchmark.** `npm run benchmark` replays the
119
+ example suite's recordings several times on the demo app (flaky
120
+ scenarios), and against deliberately broken versions of it (false
121
+ greens); `--record` also records the suite afresh with the agent and
122
+ measures the share verified on the first attempt, the share held for
123
+ review, and the time and cost. The results are published in the docs
124
+ ([benchmark](BENCHMARK.md)): today 66 of 66 replays green and no false
125
+ green on 8 broken apps.
126
+ - **A coverage map per pull request.** `saffron coverage` maps each
127
+ requirement ID (a tag matching `requirementTag`, such as `@REQ-12`) to
128
+ its scenarios, the checks their recordings make and the last run's
129
+ verdict. `--write` saves it as `.saffron/coverage.json` to commit, and
130
+ `--base origin/main` compares with the target branch's map: scenarios
131
+ added and removed, requirements no longer covered and checks a recording
132
+ no longer makes, with `--markdown` for the pull request and
133
+ `--fail-on-loss` to gate on it.
134
+ - **Side effects are counted, and can be reset.** `saffron run --plan`
135
+ marks the scenarios that change the app's data (a write request in
136
+ their recording, a unique value, or an `@side-effects` tag) and how many
137
+ times the run executes each: a recording runs once, its proof, every
138
+ stability proof and every sensitivity check again. The new `reset`
139
+ setting runs your commands before each of those fresh executions (never
140
+ before an ordinary replay), with `SAFFRON_SCENARIO`, `SAFFRON_FEATURE`
141
+ and `SAFFRON_RESET_FOR`. A failed reset stops that recording, or fails
142
+ that proof, and the run goes on.
143
+ - **Stability proofs.** `stabilityProofs` (`--stability-proofs n`) replays
144
+ a recording that passed its proof n more times, each in a fresh
145
+ execution, before it is filed. The proposal, the run summary and the
146
+ review page tell "passed one proof" from "passed the stability proofs",
147
+ and a recording that fails one gets an `unstable recording:` note, left
148
+ out of `accept --all` as the editor plugins mark it.
149
+ - **A recording plan and spending limits.** `saffron run --plan` prints
150
+ what the run would do and stops, with no browser or agent: what replays,
151
+ what records or records again (with the reason), what is refused, how
152
+ many agent sessions and proof replays that takes, and new steps several
153
+ scenarios share, to record and accept first. `--json` and the MCP tool
154
+ `run_plan` give the same as data. `maxCostUsd` (`--max-cost`) and
155
+ `maxAgentMinutes` (`--max-agent-minutes`) stop new agent sessions once a
156
+ run reaches them, and the summary splits AI cost into recording, repair
157
+ and failed attempts.
158
+ - **Every repair is checked against what its recording bound.** From the
159
+ committed recording, Saffron lists each value a step enters, each request
160
+ it sends and each check it makes, and finds the repaired action that
161
+ keeps each one. `saffron diff`, its JSON (`contract`) and the review page
162
+ show the list with that evidence. A repair that no longer sends a request
163
+ or makes a check gets a `requirement not kept:` note, and `accept --all`
164
+ leaves it out for review, as the editor plugins mark it.
165
+ - **The run summary says which run it counts.** It names the invocation
166
+ (`this run (\`saffron run ...\`, 12 scenarios, 10:04)`), splits pending
167
+ review by proof passed, failed or not run, and splits failures into the
168
+ app's and those the recorder, the agent's provider or the environment
169
+ caused (`blocked` in the report JSON). A run of named scenarios also
170
+ prints the combined report's counts. The report shows the command too.
171
+ - **A review page for proposals.** `saffron diff --html` writes
172
+ `.saffron/reports/review.html`: each Gherkin step beside the action it
173
+ records today and the one proposed, the target, what the proof captured
174
+ and any note about that step, so "enter an expiry date" becoming "check
175
+ an expiry hint" shows in the step's own row. The last run's screenshot of
176
+ the page before the agent took over is included. `saffron diff` and
177
+ `--json` show those notes under their step too.
178
+ - **Dependencies.** saffron-ai now needs `zod` 4.6.7 or later, so
179
+ upgrading it in a project with a lockfile also brings the fixed `zod`.
180
+ The other packages `npm audit` may report after an upgrade
181
+ (`body-parser`, `qs`, `hono`, `ip-address`, `fast-uri`) come through
182
+ Saffron's dependencies and are already allowed at their fixed versions:
183
+ `npm audit fix` updates them.
184
+
185
+ Editors (VS Code extension 0.7.5, JetBrains plugin 0.7.6)
186
+
187
+ - **Two more notes hold a proposal for review.** A proposal with a
188
+ "requirement not kept:" note (a repair that no longer sends a request or
189
+ makes a check its recording did) or an "unstable recording:" note (a
190
+ recording that failed an extra stability proof) is held from Accept
191
+ All, as `saffron accept --all` holds it. The new dashboard needs no
192
+ editor update: both show the runner's page.
193
+
8
194
  ## 1.2.1: Sturdier recordings, and API keys for Codex and Antigravity
9
195
 
10
196
  Released 2026-10-28.