saffron-ai 1.2.2 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md
CHANGED
|
@@ -5,7 +5,240 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
-
##
|
|
8
|
+
## 1.3.0: Recordings checked by machine, and a review every view agrees on
|
|
9
|
+
|
|
10
|
+
Released 2026-10-30.
|
|
11
|
+
|
|
12
|
+
Runner
|
|
13
|
+
|
|
14
|
+
Upgrading from 1.2.2:
|
|
15
|
+
|
|
16
|
+
- A recording that keeps part of a captured text (`pattern`) is version
|
|
17
|
+
13, which 1.2.2 and earlier refuse ("Update saffron-ai"). Upgrade every
|
|
18
|
+
machine and CI job that replays the suite together.
|
|
19
|
+
- `accept --all` now also holds proposals with a `row not proved:`, `date
|
|
20
|
+
not following the clock:`, `write not finished:` or `check changed the
|
|
21
|
+
page:` note: accept them by file once reviewed. A teammate still on
|
|
22
|
+
1.2.2 running `accept --all` takes them, so upgrade before
|
|
23
|
+
bulk-accepting. An Outline recording that fails a row it serves is filed
|
|
24
|
+
unverified, its proof error naming the row, so every runner skips it.
|
|
25
|
+
- Now refused: an Examples table that names a column twice; a text check
|
|
26
|
+
a step words as a comparison ("starts with") but records as
|
|
27
|
+
containment; under a `Then`, an action that changes the app (it is not
|
|
28
|
+
recorded, and the proposal is held); a captured credential (a login's
|
|
29
|
+
token) typed through the browser.
|
|
30
|
+
- `transport.api.headers` go only to an API under test; an unset
|
|
31
|
+
`{env:NAME}` in them, or a value that is not text, stops the request
|
|
32
|
+
instead of sending it.
|
|
33
|
+
- The run history (`.saffron/history.jsonl`) keeps its newest half once
|
|
34
|
+
it passes 64 MB.
|
|
35
|
+
- `saffron eject` refuses a capture that keeps part of its text
|
|
36
|
+
(`pattern`), which an exported test does not do yet.
|
|
37
|
+
- `--plan` counts an Outline as one recording. New output fields:
|
|
38
|
+
`estimate` (`--plan --json`), `failedRows` (`status --json`, the MCP
|
|
39
|
+
`verify` tool, the CI run view), `rows` (`diff --json`, the report
|
|
40
|
+
JSON), `invocation` (the report), `checkedAt` (a recording), and
|
|
41
|
+
`proof-N-requests.json` evidence files.
|
|
42
|
+
|
|
43
|
+
- **Recordings checked by machine, not read by people.**
|
|
44
|
+
- An Outline's recording is proved on every other row it serves, at
|
|
45
|
+
zero AI, in a fresh execution each, before it is filed. One that fails
|
|
46
|
+
on a row is filed unverified, its proof error naming the row, with a
|
|
47
|
+
`row not proved:` note; one that cannot serve a row (bound to its own
|
|
48
|
+
row's values) gets the same note, as accepted that row stays red until
|
|
49
|
+
it is recorded again. Both are held from `accept --all`. Rows whose
|
|
50
|
+
`@requires:` capability this environment lacks are not proved, and the
|
|
51
|
+
review counts them apart: "proved on 2 of 2 rows (1 refused)".
|
|
52
|
+
(SAF-159)
|
|
53
|
+
- A new recording is checked against a moved clock: every `{date}` token
|
|
54
|
+
resolved for tomorrow, the month's last day and the next month's
|
|
55
|
+
first, and the year's last day and the next year's first, and a
|
|
56
|
+
weekday, month or year left as text beside it must still match
|
|
57
|
+
(`Thursday, {date+1:MMMM D, YYYY}` does not; an amount such as
|
|
58
|
+
`2026.00` is no year). A whole date near today left as text is named
|
|
59
|
+
too, in the locale's own spellings and in the other order where only
|
|
60
|
+
one reading is a day ("13/11/2026"), unless the scenario writes it or
|
|
61
|
+
it is part of an address (`/blog/2026/10/30/`).
|
|
62
|
+
Either gets a `date not following the clock:` note, held from
|
|
63
|
+
`accept --all`. (SAF-158, SAF-165)
|
|
64
|
+
- A proof that passed while a write to the app's own site was cut off by
|
|
65
|
+
its frame moving on gets a `write not finished:` note, held from
|
|
66
|
+
`accept --all`; a replay that fails after such a write names it. The
|
|
67
|
+
app's own writes are those to the base URL, an API the run tests, or
|
|
68
|
+
the page's own site; an iframe loading, a third party's beacon, a
|
|
69
|
+
download or a 204 count for nothing, nor a read sent as a POST (a
|
|
70
|
+
GraphQL query, a search), judged by the rule a `Then`'s requests are;
|
|
71
|
+
a navigation that fails on the network counts. (SAF-170)
|
|
72
|
+
- A text check that passed on an element showing far more than the value
|
|
73
|
+
(a card) gets a `broad check:` note. (SAF-171)
|
|
74
|
+
- A page check is judged against the last step that acted on the page:
|
|
75
|
+
an API request in between cannot change the page, and named an
|
|
76
|
+
insensitive check that was not.
|
|
77
|
+
- A check of an Examples cell that is a day written out (`2026-01-05`,
|
|
78
|
+
`05-01-2026`, "Jan 5", "5th January 2026"), shown in another format
|
|
79
|
+
("05.01.2026"), binds the recording to its row, as a `{date}` cell
|
|
80
|
+
does: every row checked row 1's day. A cell such as "Order 1234" is no
|
|
81
|
+
day.
|
|
82
|
+
- `rowProofs` in saffron.config.json caps how many other rows a
|
|
83
|
+
recording is proved on; a row past it is "not run", and replays the
|
|
84
|
+
proposal on its own row later in the run.
|
|
85
|
+
- A contract re-check (only a pinned API contract changed) files the
|
|
86
|
+
recording with the Examples row, bindings, data and recording time it
|
|
87
|
+
was made with, whichever row ran the check, and records when it was
|
|
88
|
+
proved again as `checkedAt`. A re-check that would change any action
|
|
89
|
+
is red, never filed. (SAF-157)
|
|
90
|
+
- A stale reason and a row refusal start with the file and line they
|
|
91
|
+
point at (`features/plans.feature:5 step 2 reads ...`), and a held
|
|
92
|
+
note names its step's file and line after its kind. (SAF-160)
|
|
93
|
+
- **What a recording keeps.**
|
|
94
|
+
- The agent types the scenario's references (an Examples `<column>`, a
|
|
95
|
+
`{data:...}` value, a `{captured:name}` an earlier step kept) as
|
|
96
|
+
written: the browser gets the value and the recording keeps what the
|
|
97
|
+
agent wrote, as with `{env:...}`. Two references that read the same in
|
|
98
|
+
one call are left to the recording's own row matching. A captured
|
|
99
|
+
credential is never typed through the browser: the agent would read
|
|
100
|
+
it back, and the page could send it anywhere. (SAF-161, SAF-166)
|
|
101
|
+
- A value captured while recording can be typed by a later step
|
|
102
|
+
(`saffron_capture`, then `{captured:name}`). A capture can keep only
|
|
103
|
+
what a pattern picks (one group, named or not; it may hold
|
|
104
|
+
`{captured:...}`, `{date...}` and `<column>` tokens), so a booking
|
|
105
|
+
number beside its Copy label is the number alone; replay fails naming
|
|
106
|
+
the text when the pattern picks no value or two, or names one that
|
|
107
|
+
backtracks (`(a+)+`), which is stopped instead of freezing the run, as
|
|
108
|
+
a `matches` check's pattern is, and a `differs` check
|
|
109
|
+
against it compares the same part of the text. Such a recording is
|
|
110
|
+
version 13. A value the proof read with more around it (a label) gets
|
|
111
|
+
a `capture differs:` note. (SAF-166)
|
|
112
|
+
- A day the scenario writes with its month's name and no year
|
|
113
|
+
(`{date+9:D MMM}`) goes back to its token in every step, as a whole
|
|
114
|
+
date does, and so do the spellings "Nov 6 2026", "6. November 2026",
|
|
115
|
+
"2026/11/06" and, in the United States, "11/6/2026"; a path's segments
|
|
116
|
+
(`/blog/2026/10/30/`) stay as they are. (SAF-165)
|
|
117
|
+
- A `Then` checks; it never changes the app. An entry, a key press, a
|
|
118
|
+
pick, a tick, an upload, a drag, a navigation, a tab opened at an
|
|
119
|
+
address or a request that writes is not recorded under it: a write
|
|
120
|
+
request is refused before it is sent, and the proposal gets a `check
|
|
121
|
+
changed the page:` note, held from `accept --all`. A read the API
|
|
122
|
+
offers only as a POST may be sent: a GraphQL query to a `/graphql`
|
|
123
|
+
endpoint that holds no mutation (comments and strings aside), or a
|
|
124
|
+
request to an endpoint whose last segment is `search` or `query`. A
|
|
125
|
+
click or an answered dialog under a `Then` is recorded, as it may only
|
|
126
|
+
open what is checked, and held the same way, as nothing the run sees
|
|
127
|
+
can tell an opening click from one that cancels the order the check
|
|
128
|
+
reads: accept it by file once you have seen it only opens. (SAF-167)
|
|
129
|
+
- A step performed again after an attempt that did not work drops the
|
|
130
|
+
earlier attempt (`saffron_step restart`). (SAF-167)
|
|
131
|
+
- A text check the step names as a comparison (`starts with "Oct"`,
|
|
132
|
+
`ends with "-2026"`, "the same day as") is refused as plain
|
|
133
|
+
containment, naming the check that records it, in straight or curly
|
|
134
|
+
quotes or as a capitalized word (`starts with Oct`), and a pattern
|
|
135
|
+
recorded for it needs its anchor, in any group (`(?<id>^ORD\d+)`,
|
|
136
|
+
`(?i:^ORD)`) or as a lookahead (`(?=$)`). A number with a leading zero
|
|
137
|
+
("0042") is an identifier, never compared as an amount. (SAF-169)
|
|
138
|
+
- A choice opened through an element named with the row's `<column>`
|
|
139
|
+
gets a `choice opener:` note, and the agent is told to open a choice by
|
|
140
|
+
its own label, never by the value it shows. (SAF-168)
|
|
141
|
+
- **Review: every view says the same.**
|
|
142
|
+
- The review shows an Outline recording's proof row by row. One that
|
|
143
|
+
failed a row is never shown as verified: in `saffron run`, its totals
|
|
144
|
+
and cost split, the report, history and the dashboard, `saffron diff`,
|
|
145
|
+
`status --json` and the MCP tools (`failedRows`), the CI run view, the
|
|
146
|
+
pull request body and `saffron trace`, which asks before accepting it.
|
|
147
|
+
A later run that replays it keeps what its own proof found. (SAF-172)
|
|
148
|
+
- A held note stands beside the step it names in `saffron diff` and the
|
|
149
|
+
report's review tab, its file and line included, and the pull request
|
|
150
|
+
body names every note `accept --all` holds.
|
|
151
|
+
- A repair that fixes the steps before a `Then` while the `Then` still
|
|
152
|
+
fails as written, or that changes nothing and still does not replay,
|
|
153
|
+
is red, never a proposal healed again on every run.
|
|
154
|
+
- **Replay.**
|
|
155
|
+
- `ControlOrMeta`, in a key press or a click's modifiers, replays as the
|
|
156
|
+
browser's own platform has it, so a remote Linux browser driven from a
|
|
157
|
+
Mac pastes too. (SAF-175)
|
|
158
|
+
- A step on a page busy in its own script times out as it should.
|
|
159
|
+
- A popup no longer crashes the run: its first navigation has no frame
|
|
160
|
+
yet, which the network listener now allows for.
|
|
161
|
+
- A refinement after a failed proof gets the values the proof captured.
|
|
162
|
+
- **API tests.**
|
|
163
|
+
- The transport's API headers go only to an API under test, never to
|
|
164
|
+
another origin a request names. Each `{env:NAME}` is read when the
|
|
165
|
+
request is sent, and one that is not set is refused instead of sent
|
|
166
|
+
empty. A value read from `{env:...}`, or written whole for a header
|
|
167
|
+
whose name ends in a credential's word, a version aside
|
|
168
|
+
(`Authorization`, `X-Api-Key`, `X-Session-Id`, `X-Hub-Signature-256`,
|
|
169
|
+
`X-API`), is masked wherever it would show, including a response the
|
|
170
|
+
agent reads. A browser scenario's API requests carry the headers too,
|
|
171
|
+
masked in its trace, leaving out one whose `{env:...}` the job does not
|
|
172
|
+
set.
|
|
173
|
+
- A credential inside an extracted object (`extract "/data"` holding a
|
|
174
|
+
token) makes the capture a credential, and the token is masked on its
|
|
175
|
+
own, in a step that fails too. A name counts by its last word
|
|
176
|
+
(`accessToken`, `authorization`, `pwd`, `otp`, `api_key`,
|
|
177
|
+
`refresh_token`), a word that only says how it is held aside
|
|
178
|
+
(`authCode`, `apiKeyValue`, `privateKeyPem`), never one that only
|
|
179
|
+
mentions it (`sessionTitle`, `pinColor`, `total_tokens`): one rule for
|
|
180
|
+
captures, fields and headers, in replay and in an exported test alike.
|
|
181
|
+
A value of 8 characters or more counts, a password at any length, or a
|
|
182
|
+
code of 4 to 8 digits under `otp` or `pin`; a short word
|
|
183
|
+
(`"auth": "yes"`) is no secret, and masked it garbled every text it
|
|
184
|
+
appeared in.
|
|
185
|
+
- A secret of any length is masked: a value of some 7,000 characters
|
|
186
|
+
(a large token or cookie) made masking throw.
|
|
187
|
+
- A proof's API transcript is kept apart from the run's
|
|
188
|
+
(`proof-1-requests.json`), for API and browser scenarios alike. The
|
|
189
|
+
report, `saffron trace` and the CI run view show the run's own
|
|
190
|
+
requests first, and each proof's under its own name and outcome.
|
|
191
|
+
- Copies sent at once match their expected statuses in any order.
|
|
192
|
+
- **CLI, plan and reports.**
|
|
193
|
+
- A run of named scenarios folded into the last report carries its own
|
|
194
|
+
command, and says what it ran as `invocation`. (SAF-164)
|
|
195
|
+
- `--plan` estimates the browser time of its replays and recordings
|
|
196
|
+
from past runs, counts an Outline as one recording with its row
|
|
197
|
+
proofs, and counts no recording time with `--no-agent`. It counts a
|
|
198
|
+
new step's scenarios per file, and mentions `reset` as it is set.
|
|
199
|
+
(SAF-174)
|
|
200
|
+
- A suggested rename to an existing wording needs the two wordings to
|
|
201
|
+
share a word or a quoted argument, which it names, in a recording and
|
|
202
|
+
in `status` alike. (SAF-173)
|
|
203
|
+
- An Examples table that names a column twice is refused with its file
|
|
204
|
+
and line, and a file whose only Outline has no Examples rows is told
|
|
205
|
+
it needs one. While one file does not parse, the step vocabulary (the
|
|
206
|
+
LSP, MCP `search_steps`, `saffron vocab`) still offers every other
|
|
207
|
+
file's steps, and names the file that does not.
|
|
208
|
+
- `report --merge` refuses a shard whose rows or proof transcripts it
|
|
209
|
+
cannot read, a history line that is not a run is skipped, and the
|
|
210
|
+
dashboard calls a scenario flaky only when it flips in one
|
|
211
|
+
environment.
|
|
212
|
+
- The run history is read from its end, so only the runs a view needs
|
|
213
|
+
are read, and it keeps its newest half once it passes 64 MB: months of
|
|
214
|
+
CI runs made every command fail. A history that cannot be read costs a
|
|
215
|
+
run its trends, never its report.
|
|
216
|
+
- A recording with many dates is finished several times faster.
|
|
217
|
+
- `report --no-open`, and `report --run --no-open`, write the report
|
|
218
|
+
without opening a browser.
|
|
219
|
+
- `saffron run --no-agent` names the pending proposal that does not
|
|
220
|
+
replay on a row, instead of saying no recording exists, and the run
|
|
221
|
+
line no longer hides a path behind an identifier named like a secret
|
|
222
|
+
(`..._SESSION_ID`). An invalid `rowProofs` stops `--plan` too.
|
|
223
|
+
- The MCP `verify` tool takes `baseRef` only as a git ref, and says so
|
|
224
|
+
plainly otherwise; `saffron lsp --stdio` is accepted; `saffron prune`
|
|
225
|
+
says its count holds recordings, proposals and evidence folders.
|
|
226
|
+
- **Data tables for API steps.** An API request's values in a data table
|
|
227
|
+
under its step (an API key, a correlation id, query parameters, body
|
|
228
|
+
fields) record as their `<table:key>` wherever the request sends them,
|
|
229
|
+
path, headers, body or form, so editing a cell replays at zero AI. The
|
|
230
|
+
bundled skill and `saffron author` now tell the agent to keep each step
|
|
231
|
+
one sentence and put more than two values in such a table, not on one
|
|
232
|
+
long line.
|
|
233
|
+
|
|
234
|
+
Editors (VS Code extension 0.7.6, JetBrains plugin 0.7.7)
|
|
235
|
+
|
|
236
|
+
- Proposals with every new note are held, the Accept All tooltips name
|
|
237
|
+
them all, and an Outline proposal that failed a row shows as failing
|
|
238
|
+
it, in the proposal list and the CI run view.
|
|
239
|
+
- A `.saffron` file's `Fixture:` line is coloured as a block keyword, as
|
|
240
|
+
`Scenario:` is, in the editors and in the TextMate bundle the runner
|
|
241
|
+
ships.
|
|
9
242
|
|
|
10
243
|
## 1.2.2: Triage in the report, a new dashboard, and verification for coding agents
|
|
11
244
|
|