saffron-ai 1.1.0 → 1.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,362 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
5
5
  JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
6
6
  release heading names its main feature: `## <version>: <title>`.
7
7
 
8
+ ## 1.2.1: Sturdier recordings, and API keys for Codex and Antigravity
9
+
10
+ Released 2026-10-28.
11
+
12
+ Runner
13
+
14
+ - **After a review of the unreleased work.**
15
+ - An Outline row's date that the page spells out (`Friday, October 30,
16
+ 2026`) is recorded as that row's own date token, so another row never
17
+ replays on it and the same row follows the day without recording
18
+ again. A day the page works out from it (a checkout two days on)
19
+ follows the row too. A recording that kept the date as text records
20
+ once more. Such a recording is schema version 11, which 1.2.0 refuses.
21
+ Another row with a different date there fails with its reason; it
22
+ needs a Scenario of its own. A day shown in a step that does not use
23
+ the column (today's date in a banner) no longer ties the Outline to
24
+ one row.
25
+ - A date the scenario writes itself (`2026-10-30`) stays that date when
26
+ the page spells it otherwise ("Friday, October 30, 2026"), instead of
27
+ moving with the calendar from the next day on.
28
+ - A token used as a key in a whole-JSON Examples column is refused
29
+ instead of being sent as a broken field name.
30
+ - `saffron author` makes the checks a recording makes before it starts
31
+ the agent, except agy's rules for Saffron's tools, which it does not
32
+ use: with `antigravityApiKey` set and no key it refuses instead
33
+ of signing in with the Google login, and a failed session prints its
34
+ reason instead of a stack trace.
35
+ - A token outside any string in a whole-JSON Examples column
36
+ (`{"qty": {captured:count}}`) is a value of its own, so a secret
37
+ beside it can no longer add a field. A cell that is neither JSON nor
38
+ one whole token is refused instead of being filled in first. The same
39
+ holds in exported tests.
40
+ - Captured values in a proposal mask a credential an API handed out.
41
+ - A secret inside a whole-JSON Examples column can no longer rewrite
42
+ the payload.
43
+ - "Should differ" compares references and counts (`ABC-123`,
44
+ `1 night`) as text, refusing only amounts in different currencies or
45
+ a price against a label, and a brief bad reading no longer ends the
46
+ check.
47
+ - A heal that swaps a tick, untick, upload or drag for another input is
48
+ flagged as a behavior change.
49
+ - The sensitivity check gives each check a second, skips a `Then` with
50
+ no check or in a session not yet open, and cannot abort a run. The
51
+ report counts only the proposals it checked.
52
+ - `expectDisabled` is decided by the recorded element, not a fallback.
53
+ - Slashed dates are read in the locale's order, a UK date with its
54
+ weekday becomes a token, and a month the scenario names is not a
55
+ volatile locator.
56
+ - **The npm page says what Saffron is.** The package description no longer
57
+ calls it an AI-fallback test runner, its keywords cover the terms people
58
+ search for, it names its author, and the README names Antigravity among
59
+ the agents.
60
+ - **`prune --json` no longer calls a broken file's recordings orphans.**
61
+ While a feature file fails to parse, prune already deleted nothing, but
62
+ its JSON listed the hidden scenarios' live recordings as orphans. Now
63
+ they are in `orphanScan.ownershipUnknown`, beside the parse `errors`;
64
+ other files' orphans stay in `orphans`, and the JSON has the same shape
65
+ on a healthy project. `saffron status --json` splits them the same way,
66
+ where it listed no orphans at all.
67
+ - **Volatile locators are caught while recording.** A date the agent
68
+ picked that the app spells out (`October 30, 2026`, `30.10.2026`) is
69
+ recorded as a `{date}` token, as ISO dates already were, and a day the
70
+ scenario's own token wrote keeps that token. A generated id in a
71
+ selector, frame or test id, a UUID in a name, or a month heading is
72
+ refused once at `saffron_done` so the agent finds a stable target; what
73
+ is left gets a `volatile locator:` note, and `accept --all` leaves that
74
+ proposal to be accepted by file.
75
+ - **An Examples column can hold a whole JSON payload.** A request whose
76
+ body, or a part of it, is the JSON a column the step names holds is
77
+ recorded as that column, so each row sends its own payload. Before, the
78
+ payload was kept as fixed fields and every other row was refused. The
79
+ proposal names the body each row sends, and a cell that is not JSON fails
80
+ its row by name. Such a recording is schema version 10.
81
+ - **Comparisons compare like with like.** A "should differ" check between
82
+ two amounts compares the numbers, so the same price drawn in another
83
+ format has not changed. Two values of different kinds fail at once and
84
+ say why: `SEK 1,295` against `NOK 1,347` differ by currency alone, and a
85
+ whole button label against a price by its label. `saffron diff` and the
86
+ report show what each capture read in the proof replay beside its step.
87
+ A step that says an element is selectable, available or disabled records
88
+ an enabled or disabled check (`expectEnabled`, `expectDisabled`), where it
89
+ recorded visibility only; such a recording is schema version 9.
90
+ - **Recorded checks must be able to fail.** A proof shows a recording
91
+ replays, not that its checks notice anything. After a proof passes,
92
+ Saffron replays the scenario once more with no AI and runs each `Then`
93
+ on the page as it was just before the action step it follows. A check
94
+ that passes there too gets an `insensitive check:` note (picking the day
95
+ that was already picked, a price compared with a value it always had),
96
+ and `accept --all` leaves that proposal to be accepted by file. The
97
+ report shows "checks proven sensitive in N of M" beside the proofs that
98
+ passed.
99
+ - **Heals keep what the scenario tests.** A heal serves the scenario's
100
+ intent first, then the step's, and the step's wording last: it may change
101
+ how a step is done, never what the scenario exercises. The agent is told
102
+ never to replace typed data with a check of a hint, or a real payment or
103
+ sign-in route with a simulator's button, even when the final `Then`
104
+ passes, and to fail and name a missing prerequisite instead of working
105
+ around it. Saffron checks this itself: a healed step that no longer
106
+ enters, picks or uploads what the committed recording did gets a
107
+ `behavior change:` note naming what it checks instead, and
108
+ `accept --all` leaves that proposal to be accepted by file. A value moved
109
+ to another control (a text field that became a dropdown) is still
110
+ entered and raises no note.
111
+ - **iframes are recorded with their whole scope.** An element in an iframe
112
+ inside another iframe keeps both, and one of several alike iframes
113
+ (previews, cards) keeps its position, so replay acts in the iframe the
114
+ agent used. Before, an iframe picked with `first()` or `nth()` lost its
115
+ position and replay could act in another one, and an element in a nested
116
+ iframe recorded no action at all. Code the recorder cannot read, such as
117
+ an element reached through a variable or an iframe picked by a filter, now
118
+ fails the recording and names the interaction instead of being left out.
119
+ `saffron_assert` takes the same frame forms. A recording with a nested or
120
+ positioned iframe is written as schema version 8, which saffron-ai 1.2.0
121
+ and earlier ask to be updated for; others keep their version.
122
+ - **Codex and Antigravity on an API key.** A machine with no ChatGPT or
123
+ Google login, a CI job say, records and heals with Codex on
124
+ `CODEX_API_KEY` (or `OPENAI_API_KEY`) and with Antigravity on
125
+ `GEMINI_API_KEY` (or `GOOGLE_API_KEY`), billed to that API account. A
126
+ login on the machine wins over a key, so a key set for other tools never
127
+ moves your runs off your plan. Whether there is a login is asked only
128
+ when the first session starts, and only when a key is set, without
129
+ holding up other sessions; only the vendor saying you are not signed in
130
+ counts, so a timeout or a network error keeps the login. The run's first
131
+ line says how sessions will sign in, and the first session says which one
132
+ it used ("Codex: signed in with API key (CODEX_API_KEY)"); a key stored by
133
+ `codex login --with-api-key` reads as one, not as a ChatGPT login.
134
+ `saffron login codex` and `saffron login antigravity` say which sign-in
135
+ sessions will use. An expired login still counts as a login: the session
136
+ it fails says the login won over the key, and how to sign in again or use
137
+ the key. Codex's `forced_login_method` decides alone: a key never counts
138
+ as a sign-in under `"chatgpt"`, `"api"` with no key in the environment
139
+ or stored by `codex login --with-api-key` (a stored ChatGPT login does
140
+ not count) is reported as a missing key, and a refused session names the setting. Antigravity's
141
+ equivalent is `"antigravityApiKey": true` in `saffron.config.json`: the
142
+ key is used even beside an agy login, by every command, and a missing
143
+ key is reported and starts no session. A key reaches the
144
+ vendor alone: an error that quotes it, whole or cut short, shows its
145
+ variable's name, and it is never written into a recording or report.
146
+ Codex on a key is verified live; Antigravity on a key is not yet.
147
+ - **A numeric credential is never kept in a recording.** A login's token
148
+ that is only digits (a one-time code, a PIN) is written as its
149
+ `{captured:name}` in a form field, a multipart part or a JSON number
150
+ wherever it stands, whatever the field is called. Before, a field whose
151
+ name did not name an id kept the value.
152
+ - **An API is named by the words that call it.** The check that the APIs a
153
+ scenario needs are up reads a named API from its recording, or from steps
154
+ that call it one ("the orders API", "the orders service"): "I list orders
155
+ through the default API" no longer waits on a service named `orders`.
156
+ - **A changed contract the API breaks is red.** When a recording is checked
157
+ again against a changed API contract and the API no longer answers as the
158
+ new contract says, the scenario is red and the run exits 1, as before
159
+ 1.2.0, with no proposal filed (nothing was recorded again). In 1.2.0 it
160
+ was filed as an unverified proposal and the run passed. One that passes is
161
+ filed for review as before, and its run line says it replayed from the
162
+ recording, not that an agent recorded it. `saffron status` says such a
163
+ recording is checked again on the next run instead of "will re-record",
164
+ and the Saffron skill describes it.
165
+ - **A malformed `{unique:...}` stops the run before it starts.** An
166
+ unclosed token, an empty or invalid name, or one an Examples row fills in
167
+ with an empty cell (`{unique:<id>}`) is named and the run exits 2, as a
168
+ bad `{data:...}` does, instead of costing an agent session per row.
169
+ - **Failure headings name the row's values.** `Check "the customer is on
170
+ the "Gold" plan" failed`, as the detail below it already did.
171
+ - **A form field that is only a number** is written as a captured value's
172
+ token only where its name names an id or the capture, as in a JSON body:
173
+ `quantity=100` beside order 100 stays 100.
174
+ - **A status code with its own reason** names that status whatever
175
+ follows: "expect 409 Conflict and a retry hint".
176
+
177
+ Editors (VS Code extension 0.7.4, JetBrains plugin 0.7.5)
178
+
179
+ - **Proposals Accept All skips are marked.** A proposal with a "behavior
180
+ change:", "insensitive check:" or "volatile locator:" note, or one
181
+ imported from a CI run or not verified, is marked as held (VS Code:
182
+ "held for review"; JetBrains: "held from Accept All").
183
+ - **Orphans while a file fails to parse.** The Orphans view lists the
184
+ other files' orphans and groups the broken file's recordings under
185
+ "Ownership unknown until it parses"; prune waits until it parses. Needs
186
+ saffron-ai 1.2.1.
187
+ - **No more endless Loading.** A status or diff that never answers stops
188
+ after 60 seconds with its error, and later refreshes run again.
189
+
190
+ ## 1.2.0: API test automation (beta)
191
+
192
+ Released 2026-10-26.
193
+
194
+ The same plain-language scenarios now test an HTTP API, with no browser and
195
+ the workflow you know: the agent records each scenario once, every later run
196
+ replays it at zero tokens, a `Then` is never healed, and every proposal is
197
+ proved by a zero-AI replay before you review it. API test automation and
198
+ `saffron eject` are both **beta**: they work and are covered by tests, and
199
+ they are still being shaped by use, so their recordings and output may
200
+ change between releases. See "API tests" in the guide.
201
+
202
+ **Upgrading from 1.1.0.** Nothing to change in a browser project. What can
203
+ be different after `npm update`:
204
+
205
+ - A recording that holds a request is at schema version 6 or 7. saffron-ai
206
+ 1.1.0 and earlier ask to be updated for it rather than calling it
207
+ malformed.
208
+ - `@cursor/sdk` and `@openai/codex-sdk` are no longer peer dependencies.
209
+ npm never installed them, so a Codex or Cursor project installs its SDK
210
+ itself as before, and Saffron now checks its version: one missing, too old
211
+ or of another major stops the provider before it starts, naming the
212
+ command that installs a supported one
213
+ (`npm install --save-dev "@cursor/sdk@^1.0.32"`).
214
+ - A few selector forms an Outline row could pass on another row's data with
215
+ are now refused, so they run red and record again once (see "Outline
216
+ rows" below).
217
+ - A failing check now reads `Check "..." failed: <why>`, and a stale
218
+ recording says what changed, so a script that matched the old wording
219
+ needs the new one.
220
+
221
+ Runner
222
+
223
+ - **API tests (beta).** Tag a feature `@api` and its scenarios test an HTTP
224
+ API. The recording keeps each request as sent (method, path, headers, a
225
+ JSON body typed as the API expects, a form or a multipart body with a file
226
+ of the project) and the response fields later steps need, written as
227
+ `{captured:name}`. A `Then` checks the latest response: its status, its
228
+ JSON fields, a header, how long it took, or its body against the pinned
229
+ OpenAPI contract, and each check is verified before it is recorded. A
230
+ write (`POST`, `PUT`, `PATCH`, `DELETE`) is never retried, a proof replay
231
+ makes records of its own through `{unique:...}` values, and a run of
232
+ `@api` scenarios alone needs no browser installed.
233
+
234
+ - **A pinned contract, and several APIs.** `saffron api pin <url or file>`
235
+ pins an OpenAPI 3 contract (from a document URL, a Swagger UI page or a
236
+ file, JSON or YAML, following up to five redirects, each checked as the
237
+ first, a page's links read against where it landed) as
238
+ `api/openapi.json`, and a recording checked against
239
+ it goes stale when it changes. `api.services` names more APIs beside the
240
+ default one, each with its own base URL and contract; a step names the API
241
+ it calls and the recording keeps the name, never the host, so
242
+ `--api-url` points a run elsewhere. While recording, a write goes only to
243
+ a path the contract lists or the step names, and the agent hears when a
244
+ body does not match the contract's request schema.
245
+
246
+ - **Steps that wait, follow and check more.** A request polls with `until`
247
+ for a step that waits for a state (a read only, every 100 ms at the
248
+ fastest, 10 minutes at the longest), keeps a header such as `Location`, and
249
+ takes its own timeout. Field checks find an entry of a list wherever it
250
+ stands (`item`), hold for every entry (`every`), and check `count`,
251
+ `differs`, `pattern`, `min` and `max`. Every duration may be a text with a
252
+ unit and the step's tokens (`"<seconds> s"`), so each Outline row keeps
253
+ its own. The same checks work on the browser's response checks.
254
+
255
+ - **Fixtures and named operations (beta).** A `.saffron` feature tagged
256
+ `@api` may open with one `Fixture:` block, steps run once per run before
257
+ the feature's scenarios (a login, a seeded customer), recorded, proved and
258
+ accepted like a scenario. The values it keeps reach every scenario of the
259
+ feature as read-only `{captured:name}` seeds. A step set that sends a
260
+ request and keeps a value is a reusable named operation.
261
+
262
+ - **The API from a browser scenario.** A browser step can call the API
263
+ through its session's cookies, and later browser steps can type or check
264
+ what it kept.
265
+
266
+ - **Credentials the API hands out stay out of everything.** A login's token,
267
+ a session id or an API key, read from the words of the capture's name or
268
+ field, is masked as `{captured:name}` in the transcript, the report, the
269
+ trace, every error, a failed step's page URL and the agent's own view, in every spelling JSON or a
270
+ URL gives it, and is sent only to an API under test. A browser step uses
271
+ one only on the app's own origin: never in a page or frame of another
272
+ origin or a sandboxed one, never in a new tab elsewhere, a dialog's answer
273
+ or a key press with no target. While recording, the agent sends requests
274
+ only to the APIs under test, and one it sends again is written as its
275
+ token in a header, a JSON body, a form field or a multipart part; a
276
+ request never goes to a link-local address; and a multipart file is never
277
+ a hidden file such as `.env`, nor a link or hard link to one, and goes
278
+ only to an API under test. `{env:...}` secrets keep every rule they had.
279
+
280
+ - **Transcripts, reports and traces.** An API scenario leaves the requests it
281
+ sent, and the checks each step made beside them, as `requests.json`. The
282
+ report lists them per step and opens each onto its headers and bodies;
283
+ `saffron trace` replays them, the selected request and response shown
284
+ where a browser replay shows the page; nothing is drawn while a scenario
285
+ runs. Each step of a trace is titled with its own keyword (`Given`,
286
+ `And`), and the report names an Outline row by its values beside its
287
+ number. `api.transcript` (`"on-failure"`, `"on"` or `"off"`) sets how much
288
+ is kept.
289
+
290
+ - **Reviewing a recording.** `saffron diff` and proposals show what a
291
+ request sends, what it keeps and from where, and what each check expects,
292
+ so a recording can be reviewed from them. A stale recording says what
293
+ changed ("step 2 reads ... now", "the API contract ... changed"), in the
294
+ run's error and in `saffron status`. A status counts as expected only
295
+ where the step names it ("should answer 404", "returns 201 with the new
296
+ customer", "is not found"), never a quantity ("a crate with 500 units", "a product with 500 in stock",
297
+ "with 500, then").
298
+
299
+ - **Outline rows stay on their own data.** A row value in a JSON or header
300
+ `pattern` is escaped and kept whole, so `^Order <id>` for 1 never passes on
301
+ "Order 12". In selectors, a row value inside `:has(...)` or a nested XPath
302
+ predicate is checked on the element that predicate selects, text under two
303
+ negations counts as present, and a row value checked before a sibling
304
+ combinator (`+`, `~`) is refused, since the check would land on the
305
+ sibling. An untyped `differs` reads the number 5 as the text "5".
306
+
307
+ - **Export to Playwright Test (beta) covers API steps.** `saffron eject`
308
+ writes requests and their checks as calls on the exported helper, every
309
+ kind of JSON field check included (the helper carries Saffron's own
310
+ checker), with extracted values, typed bodies, forms, uploads, polling and
311
+ `api.services`. A schema check is exported with the contract's schemas
312
+ resolved into the spec and validated with ajv, as replay validates it; a
313
+ project with one installs `ajv@^8 ajv-formats@^3`, as eject says. It
314
+ guards and masks credentials as replay does, and writes as `test.fixme`,
315
+ with the reason, a scenario that reads a fixture and a browser step that
316
+ carries a captured credential.
317
+
318
+ - **Step reuse with the API.** A value the agent kept while recording reaches
319
+ the reused steps after it, so a new scenario built mostly from known steps
320
+ records without going back to the agent. A reused step whose recording no
321
+ longer matched is reported as re-recorded only when the agent did record
322
+ it again.
323
+
324
+ - **No agent session that could not help.** When the app or API under test
325
+ does not answer at all, a scenario that would record or heal is red with
326
+ "no agent session was started" and why, instead of spending tokens on a
327
+ server that is not there (browser scenarios too). Only the APIs a scenario
328
+ names are checked, so an unrelated one being down blocks nothing; any port
329
+ is checked; behind a proxy no check is made. An API `Then` that fails
330
+ after every request answered as recorded is red at once, since a `Then` is
331
+ never healed. A changed contract is checked again by a zero-AI replay of
332
+ the same recording and filed for review, instead of being recorded again.
333
+ A run with `--no-agent` says so where it needed the agent.
334
+
335
+ - **Reviewing and reading results.** `saffron diff` takes a feature file
336
+ for its pending proposals, and the run's hint says to review with it. A
337
+ failure and a transcript check name an Outline row's values instead of its
338
+ `<column>`, and a null check reads "equal to null". Chronic advice counts
339
+ only reds that cost a session, for scenarios the run ran, and the "vs
340
+ previous run" line names scenarios that turned green. The agent counts
341
+ steps from 1 in what a person reads, as the runner does.
342
+
343
+ - **`saffron init` ignores `.env`.** The `.gitignore` entries it writes
344
+ include `.env`, where `{env:...}` secrets are kept.
345
+
346
+ - **Agents know about API tests.** `saffron mcp` has an `api_contract` tool
347
+ that lists every API's operations, the Saffron skill covers `@api`
348
+ scenarios (`references/api.md`), and the instructions `saffron init`
349
+ writes and the Claude Code and Codex plugins describe them.
350
+
351
+ - **Internal.** The runner is split along the line a second kind of test
352
+ crosses: the orchestrator owns what every medium shares (freshness,
353
+ proposals, proofs, heals and the verdict) and drives the browser and the
354
+ HTTP medium through one contract. Nothing changes for a project.
355
+
356
+ Editors (VS Code extension 0.7.3, JetBrains plugin 0.7.4)
357
+
358
+ - **An API scenario's requests in Last Run.** Last Run never opens an API
359
+ scenario's request transcript as a screenshot: only failure and
360
+ before-heal pictures open as one, as the runner picks them. Such a row
361
+ offers Open Replay, which steps through the requests it sent. Needs
362
+ saffron-ai 1.2.0.
363
+
8
364
  ## 1.1.0: Secrets the AI agent never sees
9
365
 
10
366
  Released 2026-10-22.
package/README.md CHANGED
@@ -4,7 +4,7 @@
4
4
 
5
5
  Free to use · [npm: saffron-ai](https://www.npmjs.com/package/saffron-ai) · [site](https://saffron-ai.io)
6
6
 
7
- Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude by default, or Codex, Google AI or Cursor on the subscription you already have, driving Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**. A human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
7
+ Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude by default, or Codex, Antigravity or Cursor on the subscription you already have, driving Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**: a human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
8
8
 
9
9
  When the UI changes and a cached step fails at runtime, the agent takes over mid-execution with full scenario context, adapts, and finishes the run, then files a **cache proposal** you accept or reject, like a snapshot test update. Reports state exactly what was adapted and what it cost.
10
10
 
@@ -17,7 +17,7 @@ When the UI changes and a cached step fails at runtime, the agent takes over mid
17
17
  - **Verified proposals.** Every recording/heal is proof-replayed zero-AI before it's filed (with one bounded refinement pass on failure), so proposals arrive stamped `verified ✓` or honestly `UNVERIFIED ✗`.
18
18
  - **Honest cost reporting.** Reports break out prompt-cache reads/writes (the real bill of agent sessions) next to the in+out token count, per scenario and in totals, and name the models used. On an API key the dollars are what was billed; on a Claude subscription the headline is the 5-hour plan window before and after the run, with the dollars as the API-equivalent.
19
19
  - **Your subscription, your choice of agent.** Recording and healing run on the plan already logged in on the machine: Claude (default), Codex on a ChatGPT plan, Google AI Pro/Ultra through the Antigravity CLI (experimental), or Cursor. Set `"provider"` in the config; each one runs through its vendor's official client, and every provider records through the same recorder, so the caches are identical in kind.
20
- - **Trend memory.** Every run appends to `.saffron/history.jsonl`; reports show deltas vs the previous run and 20-run sparklines, and Saffron flags **chronic scenarios** (healing repeatedly. Re-record instead of paying again) plus recurring failure themes.
20
+ - **Trend memory.** Every run appends to `.saffron/history.jsonl`; reports show deltas vs the previous run and 20-run sparklines, and Saffron flags **chronic scenarios** (healing repeatedly: re-record instead of paying again) plus recurring failure themes.
21
21
 
22
22
  ## Step reuse: new scenarios get cheaper as your suite grows
23
23
 
@@ -29,6 +29,19 @@ Like a maturing Cucumber suite, most of a new feature file is steps you already
29
29
 
30
30
  Data tables and doc strings are first-class: 2-column key/value tables parameterize the recording (`<table:username>`), multi-row record tables parameterize per cell (`<table:1:firstName>`), and `"""` doc strings record as `<docstring>`, so editing *values* or *content* replays at zero tokens, while structural changes (keys, headers, row counts) honestly re-record. Unambiguous params are scenario-wide, so a later assertion on a note's text follows content edits too. **Secrets** stay out of everything: `{env:VAR}` resolves from the environment (or a git-ignored `.env`) at replay, recordings and reports are masked back to the token, and missing variables fail fast by name.
31
31
 
32
+ ## API tests (beta)
33
+
34
+ The same plain-language scenarios test a REST/JSON API with no browser: tag a
35
+ feature `@api` and the agent records each step as requests with typed bodies
36
+ and kept values, and checks of status, fields, headers, timing and the
37
+ schema of a pinned OpenAPI contract (`saffron api pin`). Replay sends the
38
+ requests again at zero tokens, a write never twice, and a changed contract
39
+ makes the recordings that checked it stale. `Fixture:` blocks set up once per
40
+ feature, browser scenarios can call the API too, and secrets and the
41
+ credentials an API hands out stay out of recordings, reports and traces.
42
+ See "API tests" in the [guide](https://saffron-ai.io/docs). API test
43
+ automation is in beta: it works, and it is still being shaped by use.
44
+
32
45
  ## Install
33
46
 
34
47
  ```bash
@@ -73,13 +86,14 @@ node dist/cli/index.js -p examples accept --all
73
86
 
74
87
  | Command | What it does |
75
88
  |---|---|
76
- | `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b[,b...]] [--device name] [--viewport WxH] [--no-hooks] [--workers n] [--agent-workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
89
+ | `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b[,b...]] [--device name] [--viewport WxH] [--no-hooks] [--workers n] [--agent-workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode] [--api-url u \| name=u]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`, point the APIs at another environment with `--api-url`. Exit 1 on red (and on yellow with `--strict`). |
77
90
  | `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
78
91
  | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
79
92
  | `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
80
93
  | `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config, and suite quality from the run history (failing, flaky, never run, slowest, trends, per tag). |
81
94
  | `saffron steps [search] [--json] [--snippets]` | List/search the step vocabulary (files + caches) with recorded/divergent/unrecorded badges and usage; `--snippets` writes `.vscode/saffron.code-snippets` for native VS Code completion. |
82
- | `saffron mcp` | Serve the vocabulary and the project overview to AI assistants over stdio MCP (`search_steps`, `list_step_sets`, `project_status`), e.g. `claude mcp add saffron -- npx saffron mcp`. |
95
+ | `saffron mcp` | Serve the vocabulary, the project overview and the pinned API contract to AI assistants over stdio MCP (`search_steps`, `list_step_sets`, `project_status`, `api_contract`), e.g. `claude mcp add saffron -- npx saffron mcp`. |
96
+ | `saffron api pin <url or file> [--out file] [--service name]` | Pin an API's OpenAPI 3 document in the project (`api/openapi.json` by default) for `@api` scenarios' schema checks: from a Swagger UI page, the document's URL, or a file. Then set `"api": { "contract": "api/openapi.json" }` in `saffron.config.json`. `--service orders` pins a second API's contract, named under `api.services`. |
83
97
  | `saffron author <prose-file> [-o out]` | Draft a `.saffron` feature file from plain-paragraph requirements, reusing the project's step vocabulary (AI; reports how many lines are seedable). |
84
98
  | `saffron lsp` | Run the Saffron language server over stdio for JetBrains (LSP4IJ/Ultimate), Neovim, any LSP editor: badge completion, StepSet go-to-definition, hover, diagnostics. Highlighting for JetBrains comes from the bundled TextMate grammar (`textmate/saffron` in the package); run `npx saffron lsp` in a terminal for setup steps. |
85
99
  | `saffron init [--agents list] [--examples]` | Make the project agent-ready (`--examples` installs the Saucedemo example suite): installs the bundled **Saffron agent skill** into `.claude/skills` and `.agents/skills` (Copilot/Cursor via `--agents`), registers the `saffron mcp` server in `.mcp.json` / `.cursor/mcp.json` / `.vscode/mcp.json`, adds a managed block to `AGENTS.md`/`CLAUDE.md`, scaffolds config. |
@@ -114,10 +128,10 @@ Configuration lives in `saffron.config.json` at your project root:
114
128
  ```
115
129
 
116
130
  `provider` (also `--provider`) picks whose agent records and heals:
117
- `claude` (default), `codex` (ChatGPT plan; `npm i -D @openai/codex-sdk`),
118
- `antigravity` (Google AI Pro/Ultra through Google's `agy` CLI,
131
+ `claude` (default), `codex` (ChatGPT plan or `CODEX_API_KEY`; `npm i -D "@openai/codex-sdk@>=0.158.0"`),
132
+ `antigravity` (Google AI Pro/Ultra or `GEMINI_API_KEY`, through Google's `agy` CLI,
119
133
  experimental; `npx saffron login antigravity` once) or `cursor`
120
- (`npm i -D @cursor/sdk`, then `npx saffron login cursor`). `model` takes
134
+ (`npm i -D "@cursor/sdk@^1.0.32"`, then `npx saffron login cursor`). `model` takes
121
135
  that provider's model names. `npx saffron login` says what a provider
122
136
  still needs on this machine. Every sign-in flow is in
123
137
  [AI providers and sign-in](https://saffron-ai.io/docs/ai-providers).
@@ -153,9 +167,8 @@ features/ your .saffron and .feature files
153
167
  The roadmap is at **[saffron-ai.io/docs/roadmap](https://saffron-ai.io/docs/roadmap)**:
154
168
  milestones, statuses and decision gates, and everything shipped so far in
155
169
  its "Where we are" table.
156
- Headlines ahead: API testing (reusable REST operations, fixtures and schema
157
- validation), failure video, LLM-less MCP replay, the Level-2 shared step library
158
- (gated on Level-1 divergence data), and the long-term open-core SaaS layer.
170
+ Headlines ahead: failure video, LLM-less MCP replay, the Level-2 shared step
171
+ library (gated on Level-1 divergence data), and the long-term open-core SaaS layer.
159
172
 
160
173
  ## Documentation
161
174