saffron-ai 0.9.8 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,13 +1,14 @@
1
1
  {
2
2
  "name": "saffron-ai",
3
- "version": "0.9.8",
3
+ "version": "1.0.0",
4
4
  "description": "AI-fallback test runner for end-to-end tests written in plain language: zero-token cached replay, runtime AI healing, honest reports",
5
5
  "scripts": {
6
6
  "test": "vitest run",
7
7
  "build": "tsc",
8
8
  "dev": "tsc --watch",
9
9
  "e2e": "node scripts/e2e.mjs",
10
- "prepublishOnly": "npm run build && npm test && npm run bundle",
10
+ "prepublishOnly": "npm run build && npm test",
11
+ "prepack": "npm run bundle",
11
12
  "docs": "node scripts/build-docs.mjs && node scripts/build-site-docs.mjs && node scripts/build-claude-plugin.mjs && node scripts/build-vscode-keywords.mjs",
12
13
  "bundle": "esbuild src/cli/index.ts --bundle --platform=node --format=esm --packages=external --minify --legal-comments=none --outfile=dist-pkg/cli.js",
13
14
  "plugin": "node scripts/build-claude-plugin.mjs"
@@ -65,7 +66,7 @@
65
66
  "url": "https://github.com/s-chathuranga-j/saffron-community/issues"
66
67
  },
67
68
  "engines": {
68
- "node": ">=20.6"
69
+ "node": ">=22.12"
69
70
  },
70
71
  "files": [
71
72
  "dist-pkg",
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: saffron
3
- description: Write, run, and maintain end-to-end UI tests with Saffron, where tests are written in plain language, an AI agent records each scenario once, and every later run replays at zero tokens. Use when creating or editing .feature / .saffron files, defining StepSets, reviewing .saffron/cache proposals, running `saffron run`, or when the user mentions Saffron, Gherkin scenarios, step sets, cached replay, or `saffron.config.json`.
3
+ description: Create, translate, or edit Saffron UI test scenarios in .saffron or .feature files, including validating their recordings and replay. Use when asked to author or fix these test files in a Saffron project. Do not activate for ordinary application development, generic Cucumber projects, or requests only to run existing tests, inspect reports, review proposals, or configure Saffron.
4
4
  license: Saffron Free Use License v1.0 (see the saffron-ai package LICENSE)
5
5
  metadata:
6
6
  author: saffron-ai
@@ -9,8 +9,19 @@ metadata:
9
9
 
10
10
  # Saffron
11
11
 
12
- Saffron runs plain-Gherkin scenarios in a real browser. There are **no
13
- step definitions and no glue code**: on the first run an AI agent performs
12
+ Apply this workflow only when asked to create, translate, or edit Saffron
13
+ scenarios in `.saffron` or `.feature` files. A Saffron mention or an
14
+ existing test file in the project does not activate authoring work.
15
+ The run, review, and configuration commands below support that authoring
16
+ task; they are not instructions to change unrelated tests or settings.
17
+ Respect draft-only or no-execution requests and report the remaining
18
+ validation explicitly.
19
+
20
+ Saffron runs end-to-end tests written in plain language, in a real
21
+ browser. A `.saffron` file reads like Given/When/Then and goes further:
22
+ StepSets, several people in one scenario (`As "buyer"`), data tokens and
23
+ data-driven outlines; a plain `.feature` file runs as it is. There are
24
+ **no step definitions and no glue code**: on the first run an AI agent performs
14
25
  each step and records what it did into a JSON cache; every later run
15
26
  is replayed by Saffron in a real browser: zero AI calls, zero tokens, at
16
27
  the speed of the browser and the app. (The cache is Saffron's own JSON, run
@@ -36,9 +47,11 @@ npx saffron steps "login" # search
36
47
  npx saffron steps --json # machine-readable
37
48
  ```
38
49
 
39
- Prefer a `●` recorded wording verbatim. Quoted values may differ freely
50
+ Prefer a `●` recorded wording verbatim. Quoted values may differ
40
51
  (`I enter "admin"` seeds from `I enter "bob"`), so parameterize with
41
- quotes rather than inventing new phrasings. `saffron init` registers the
52
+ quotes rather than inventing new phrasings. A recording that still
53
+ names the old value elsewhere (a `delete-bob` test id, a `#row-bob`
54
+ selector) is not reused for the new one: that step records afresh. `saffron init` registers the
42
55
  `saffron` MCP server with Claude Code, Cursor and VS Code; its
43
56
  `search_steps` / `list_step_sets` / `project_status` tools give the same
44
57
  answers without a shell.
@@ -55,52 +68,149 @@ cached (editing their text re-records them), which proposals await
55
68
  review, and whether the vocabulary has divergent steps or duplicate
56
69
  wordings to converge before adding more.
57
70
 
58
- ## 2. `Then` steps are sacred: write them as the verdict
71
+ ## 2. `Then` steps are the verdict
59
72
 
60
- Saffron never heals an assertion: the agent may help *reach* a `Then`,
61
- never make it pass. So `Then` lines must state exactly what must be true,
62
- with stable text, never volatile values (prices, dates, counters). A `Then`
73
+ Final `Then` steps must check the state reached by the actions. Never
74
+ change guests, rooms, filters, or other application state inside an
75
+ assertion to manufacture its expected result. Mid-scenario assertions
76
+ may adapt only under the project's explicit policy; final assertions
77
+ remain strict. State exactly what must be true using stable text or
78
+ named captures for dynamic values (prices, dates, counters). A `Then`
63
79
  that records no check at all fails on replay, so every assertion step must
64
80
  end up with something recorded against it. Put
65
81
  the final assertions **last**; the trailing block of `Then` steps is
66
- strict under every policy. Use `Given`/`When` for actions.
82
+ strict under every policy. Use `Given`/`When` for actions. A text check
83
+ passes only on text the page shows: hidden elements, hidden parts of an
84
+ element and hidden dropdown options never count, so "should be shown"
85
+ means visible.
67
86
 
68
87
  ## 3. Never bake volatile or secret values into files
69
88
 
70
89
  - Secrets: `{env:VAR}`, as in `When I enter "{env:ADMIN_PASSWORD}" in the password field`.
71
90
  - Shared test data (accounts, names, enum values): put it in `data/*.json` and write `{data:users.admin.email}` (file, then dotted path; one value, not a list). The recording stores the token, so editing the file changes the next replay at zero tokens. Never for secrets: data files are committed. To check a whole list (an enum), point a Then step at it: `Then the status filter should list every {data:enums.OrderStatus}`; replay iterates the file's list, so adding a value needs no re-record. `Examples: {data:roles}` reads outline rows from `data/roles.csv` or a JSON list (`.saffron` only). A value that must be new on every run (an email for a sign-up) is `{unique:name}`: fresh per run, the same within it.
72
91
  - State the application must be in before the suite (seeded users, a reset database): do not script it through the UI in every scenario. The project's `saffron.config.json` takes `"setup"` and `"teardown"` shell commands, run once around `saffron run` (teardown always runs). Propose a seed script there; keep scenario steps for the behaviour under test.
73
- - Dates: write intent, not literals: "1 day from today" records as `{date+1}`.
92
+ - Dates: write intent when the day should move ("1 day from today" records as `{date+1}`); a date you write out (`I enter 2026-10-10`, in a table, a data file or `{env:...}`) is entered as written on every run. An Examples cell such as `{date+2:DD.MM.YYYY}` gives each row its own day.
74
93
  - Dynamic display values: capture and compare, e.g. `I record the total as "first"` … `"first" should differ from the displayed total`.
75
94
 
76
- ## Writing a scenario: the shape that records cleanly
95
+ ## Authoring workflow
96
+
97
+ 1. **Survey once.** Read the relevant scenarios and project config, then
98
+ `saffron status --json` (or `project_status`). Search the vocabulary
99
+ with `saffron steps "<words>"` / `search_steps` and check `list_step_sets`.
100
+ Reuse recorded wording only when it expresses the same behavior.
101
+ 2. **Inspect the behavior.** For a new or uncertain flow, use the available
102
+ browser tools to walk the relevant actions and observe their results.
103
+ Check the actual labels, initial state, comparison target, and loading
104
+ signal. Exercise each transition you will assert; seeing the controls
105
+ alone is insufficient. Existing tests and source code help locate the
106
+ flow, but do not establish its current UI behavior. Reuse recent,
107
+ relevant browser evidence instead of exploring the whole application.
108
+ If access or test data is missing, identify the gap and produce a draft
109
+ with explicit assumptions; do not invent observations.
110
+ 3. **Design one behavior per scenario.** When translating Cypress or other
111
+ tests, preserve the intended coverage, reconsider assumptions, and
112
+ split independent changes. Compare the same target immediately before
113
+ and after the change under test. Do not assume every intermediate value
114
+ must differ from every earlier value. Keep required setup and waits;
115
+ there is no fixed step-count limit.
116
+ 4. **Write to the contract below.** Add to the feature for that area. In
117
+ `.saffron`, reuse StepSets for repeated flows; `.feature` uses standard
118
+ Gherkin without StepSets or file-backed Examples. Resolve syntax and
119
+ data diagnostics and check existing wording before recording.
120
+ 5. **Record the affected scenarios.** Select each new or changed scenario
121
+ by line or name, for example `saffron run features/x.saffron:<line>`.
122
+ If a shared StepSet changes, include its affected callers. Avoid a
123
+ whole-suite recording or disabling reuse by default. Re-record a faulty
124
+ recording only after diagnosing its failed step and proof evidence.
125
+ 6. **Review what was recorded.** Use `saffron diff` and the report/trace.
126
+ Check that captures and comparisons use the same names and intended
127
+ elements, waits observe the real signal, and final assertions contain
128
+ no state-changing actions or replacement baselines. A `verified ✓`
129
+ proposal proves replay passed; it does not prove the recording faithfully
130
+ tests the written requirement. Investigate `UNVERIFIED` proposals; do
131
+ not accept them to finish the task or weaken the expected behavior.
132
+ 7. **Prove completion.** Accept only the reviewed, verified proposals for
133
+ this task, then run the affected scenarios with `--no-agent`. Complete
134
+ means those scenarios pass with zero AI calls and zero tokens, with
135
+ their intended assertions preserved. Report exactly what passed and
136
+ any scenarios or prerequisites not checked. If acceptance is reserved
137
+ for someone else or execution is unavailable/excluded, deliver the
138
+ draft or pending proposals and name the unfinished validation. Keep
139
+ feature files, shared data, and accepted caches together for review;
140
+ commit only when the task or project workflow calls for it.
141
+
142
+ ## Writing steps that record reliably
143
+
144
+ - **One clear intent per step.** Use visible labels, roles, and names:
145
+ `the "Save" button`, `the email field`. Keep CSS selectors, XPath, test
146
+ ids, and component names out of scenario prose. Granularity means an
147
+ independently understandable action or check, not necessarily one click.
148
+ - **Name captured values explicitly.** Write `I record the displayed price
149
+ for the selected date as "beforePrice"`, then `the displayed price for
150
+ the selected date should differ from "beforePrice"`. Reuse the exact
151
+ name; avoid `the price I noted` or `the previous value`. Record a baseline
152
+ before the action, and never recreate it inside the final assertion.
153
+ - **Keep the target and value comparable.** Compare the same date, item,
154
+ or row across the change. If testing an amount change, capture amount-only
155
+ text, not a whole button label containing a date, currency, or guest count.
156
+ Saffron's capture stores text; calling it "numeric" does not add numeric
157
+ parsing or arithmetic. Do not request unsupported calculations as though
158
+ they were available replay operations.
159
+ - **Wait for an observed result.** State the relevant visible state or API
160
+ response instead of `finished loading` without a signal or a fixed sleep.
161
+ A currency option called `GBP` may display `£` on prices. Use the observed
162
+ display value for the price wait; a changed selector alone does not prove
163
+ prices refreshed. Do not wait for the expected assertion outcome merely
164
+ to make it pass; the final check must still establish the requirement.
165
+ - **Keep test inputs explicit.** Use the data, secret, and date rules above.
166
+ Establish the starting guests/rooms or other relevant state. Name required
167
+ seeded data instead of silently assuming it exists. Conditional actions
168
+ such as `accept cookies if prompted` record a no-op when absent; that is
169
+ not a promise that replay will discover every future branch.
170
+
171
+ For example, after selecting a date and observing its price:
77
172
 
78
173
  ```gherkin
79
- Feature: Checkout
80
-
81
- Background:
82
- Given I open the application
83
- And I accept cookies if prompted # conditional steps are fine
84
-
85
- @smoke
86
- Scenario: Guest can complete checkout
87
- When I add "Blue Widget" to the cart
88
- And I place the order and wait for the order API to return 201
89
- Then I should see "Order placed"
90
- And the order request should have returned 201
174
+ Scenario: Changing two adults to one updates the selected date's price
175
+ Given the rooms and guests button shows "1 room, 2 adults"
176
+ When I record the displayed calendar price for the selected date as "beforePrice"
177
+ And I open the rooms and guests panel
178
+ And I reduce the adults in room 1 from 2 to 1
179
+ And I wait for the calendar price refresh indicator to disappear
180
+ Then the rooms and guests button should read "1 room, 1 adult"
181
+ And the displayed calendar price for the selected date should differ from "beforePrice"
91
182
  ```
92
183
 
93
- Guidelines the recorder rewards:
184
+ Use this shape only where the app actually has that refresh indicator and
185
+ pricing rule. Setup must select a date with suitable test availability.
186
+ An independent child or second-room price change belongs in its own
187
+ scenario with its own baseline. For currency changes, distinguish the
188
+ selector's code from the observed display symbol and capture amount-only
189
+ text. See [syntax.md](references/syntax.md#a-complete-file) for a full file.
94
190
 
95
- - One intent per step; 5–10 steps per scenario. Long scenarios heal badly.
96
- - Name the page or control in the step (`on the checkout page`, `the "Save" button`).
97
- - Data belongs in tables and doc strings, not in prose (see references).
98
- - Conditional wording (`if prompted`) records as a no-op when absent.
99
- - Real-world furniture just works if you say it: dialogs ("and confirm the
100
- dialog"), uploads ("upload the file "x.txt" as the attachment"),
101
- drag-and-drop ("drag the card onto the done column"), iframes
102
- (interact normally), network waits ("wait for the order API to return
103
- 201", "wait until the job status API reports "READY"").
191
+ `saffron author` drafts from supplied requirements and vocabulary without
192
+ browser inspection. Include observed labels, waits, prerequisites, and
193
+ expected behavior in its input. Its output must still go through steps
194
+ 5 to 7 above before it is described as a working test.
195
+
196
+ ## Several people in one scenario: `As "name"`
197
+
198
+ ```gherkin
199
+ Given As "seller" I am on the orders page
200
+ When As "buyer" I order the blue lamp # the buyer's own browser
201
+ Then As "seller" I see an order for the blue lamp
202
+ And the order is marked new # still the seller
203
+ ```
204
+
205
+ Each name is its own browser session (cookies, login, tabs). It carries
206
+ over to the next steps until another `As "..."`; `As "default"` goes
207
+ back. Give each person a login with `"sessions": { "buyer": {
208
+ "storageState": ".auth/buyer.json" } }`; a person without an entry
209
+ starts from the top-level `storageState` (two such people start as the
210
+ same user, and the run warns). Keep a step's `As "..."` when rewording
211
+ it, and never copy one person's prefix onto another's step: a recording
212
+ is reused only within its own session, so the buyer's typed address
213
+ never seeds the seller's step.
104
214
 
105
215
  ## Repeated step groups → StepSets (`.saffron` files only)
106
216
 
@@ -128,60 +238,45 @@ Scenario: Checkout happy path
128
238
  `Feature:` header; it yields no runnable scenarios).
129
239
  - Start a set with a guard step, end it with an exit assertion.
130
240
 
131
- ## The workflow
241
+ ## Commands for completing an authored test
132
242
 
133
243
  ```bash
134
- npx saffron run # record misses, replay hits, heal failures
135
- npx saffron run --filter @smoke # by tag; or an expression: "@e2e and not @wip", "(@smoke or @sanity) and not @flaky"
136
- npx saffron run --no-agent # replay only (CI without AI access)
137
- npx saffron run features/login.saffron:12 # one scenario: any line in it (an Examples row runs that row)
138
- npx saffron run "features/login.saffron:Successful login" # one scenario, by name
139
- npx saffron status # what is cached, pending, tagged; vocabulary health
140
- npx saffron diff # what a proposal changes, action by action: the evidence
141
- npx saffron accept # list pending proposals with narratives
142
- npx saffron accept --all # promote verified proposals to caches
143
- npx saffron accept <file> [<file>...] # promote chosen ones
144
- npx saffron reject <file> [<file>...] # discard chosen ones; the agent retries next run
145
- npx saffron report # open the HTML report
146
- npx saffron dashboard # suite quality: failing, flaky, never run, slowest, per tag
147
- npx saffron run --rerecord --filter @t # a recording is wrong: record it fresh
148
- npx saffron trace "<scenario>" # a red or healed scenario, step by step, with the page at every action ("trace" on in config)
149
- npx saffron runs # this branch's CI runs; a heal made there: saffron import --run <id>
150
- npx saffron report --run <id> # a CI run's results without importing (kept in .saffron/ci/)
244
+ npx saffron run features/login.saffron:12 # record/replay one affected scenario
245
+ npx saffron diff # inspect pending recordings
246
+ npx saffron trace "<scenario>" # inspect a failed or adapted step (trace enabled)
247
+ npx saffron accept <proposal-file> # accept a reviewed, verified proposal
248
+ npx saffron run features/login.saffron:12 --no-agent # prove cached replay
249
+ npx saffron status --json # confirm affected scenario state
151
250
  ```
152
251
 
153
- Results: **green** = cached pass · **yellow** = pending review (an AI
154
- recording or adaptation filed a proposal) · **red** = failed. Proposals
155
- arrive stamped `verified ✓` (zero-AI proof replay passed) or
156
- `UNVERIFIED ✗`; accept the former, investigate the latter. A proposal is
157
- bound to the scenario and the committed recording as they were when it was
158
- filed: `saffron accept` refuses a STALE one (the step or the recording
159
- moved since) with no override; run the scenario again for a fresh one. Commit
160
- `.saffron/cache/` like snapshots; git-ignore `.saffron/reports/`.
161
-
162
- Cost in the summary line and report: on an API key the dollars are what
163
- was billed; on a Claude subscription (Claude Code login, no key) the
164
- headline is the 5-hour plan window before and after the run, and the
165
- dollars are the API-equivalent, marked `≈`. Both name the models used.
166
-
167
- Scenario Outlines record once: the first Examples row records, later
168
- rows replay that proposal at zero AI. Write the `Then` with the row's
169
- value quoted exactly (`Then I should see "<message>"`); the recorder
170
- keeps the placeholder, so every row satisfies it.
171
-
172
- **Never hand-edit files under `.saffron/cache/` or `.saffron/proposals/`**
173
- unless deliberately authoring a manual cache (`"recordedBy": "manual"`).
174
- To change behavior, change the `.feature`/`.saffron` text and let the run
175
- re-record (mostly seeded from existing step recordings).
176
-
177
- **Review a CI heal.** A heal made in CI travels as a bundle (`saffron
178
- export` in the workflow, uploaded as an artifact). On the reviewer's
179
- checkout, `saffron runs` lists the branch's CI runs and `saffron import
180
- --run <id>` (or `saffron import <bundle>`) files the run's proposals,
181
- report and traces here, each checked against this checkout; then
182
- `saffron diff`, `saffron accept` and commit `.saffron/cache/` as usual.
183
- With `saffron export --branch <name>` CI pushes the proposals as a branch
184
- with a draft pull request instead: switch to it, accept, commit, push.
252
+ Results: **green** = cached pass · **yellow** = pending review · **red** =
253
+ failed. `verified ✓` means the proposal's zero-AI proof replay passed;
254
+ review its meaning before accepting. `UNVERIFIED ✗` needs investigation.
255
+ A STALE proposal belongs to an older scenario or recording: run the
256
+ scenario again for a fresh one. Keep accepted `.saffron/cache/` files
257
+ with the test; ignore `.saffron/reports/`.
258
+
259
+ For outlines, check every affected Examples row with `--no-agent`, not
260
+ only the row used to record. Quote the row's expected value in `Then`
261
+ (e.g. `Then I should see "<message>"`) so the recording keeps the parameter.
262
+ One recording serves every row, and it cannot tell whether a row value
263
+ glued inside other text belongs to the row (an email built from `<user>`,
264
+ `INV20261013`, `orders-US`) or a field lists every row's value. Write
265
+ `<column>` in every step whose text or typed value depends on the row;
266
+ otherwise the other rows fail red naming the column, and some cases
267
+ (dates without a year, two-letter values, blank cells) are not caught at
268
+ all. Keep a check that lists all rows' values out of the Outline.
269
+ Tests live in the folder named by `features` in `saffron.config.json`
270
+ (`features/` by default).
271
+
272
+ **Never hand-edit generated `.saffron/cache/` or `.saffron/proposals/` to
273
+ make a test pass.** Change incorrect scenario text and record again, or
274
+ use `saffron run <scenario-selector> --rerecord` when the recording is wrong.
275
+ Inspect failures before another paid recording; report an application,
276
+ fixture, or unsupported-operation blocker instead of repeatedly retrying
277
+ or weakening assertions. For CI artifacts, browser matrices, provider
278
+ setup, and other configuration needed by the requested test, read
279
+ [Configuration & CI](references/config.md).
185
280
 
186
281
  ## Anti-patterns that cost money or hide bugs
187
282
 
@@ -189,10 +284,10 @@ with a draft pull request instead: switch to it, accept, commit, push.
189
284
  |---|---|
190
285
  | Invent a synonym for an existing step | Reuse the `●` wording from `saffron steps` |
191
286
  | Assert inside a `When` ("When I see the dashboard") | Act in `When`, assert in `Then` |
192
- | `Then the price is "NOK 1,148"` | `Then the price should match "NOK [\d,]+"` intent, or capture + compare |
287
+ | Hard-code a changing price observed during authoring | Capture and compare for a change; use an exact price only when the requirement fixes it |
193
288
  | Put a password literal in a step | `{env:VAR}` |
194
289
  | Sleep/wait "for 3 seconds" | Wait for a UI signal or an API response |
195
- | One 30-step scenario | Several 5–10 step scenarios + StepSets |
290
+ | Several independent changes and all-pairs comparisons in one scenario | One behavior and its own baseline per scenario; reuse setup with StepSets in `.saffron` |
196
291
  | Edit cache JSON to fix a test | Edit the scenario text and re-run, or `saffron run --rerecord` if the recording itself is wrong |
197
292
 
198
293
  ## References
@@ -200,4 +295,5 @@ with a draft pull request instead: switch to it, accept, commit, push.
200
295
  - [Syntax cheat sheet](references/syntax.md): tables, doc strings,
201
296
  outlines, secrets, dates, network waits, page furniture, StepSets.
202
297
  - [Configuration & CI](references/config.md): `saffron.config.json`,
203
- flags, `--strict`, cross-browser, workers, heal model, secrets setup.
298
+ flags, `--strict`, cross-browser and browser matrix, devices, workers
299
+ and agent workers, heal model, secrets setup.
@@ -47,6 +47,7 @@ your-project/
47
47
  | `dataDir` | Folder of JSON files read by `{data:file.key}` tokens (default `data`). Committed; not for secrets |
48
48
  | `baseURL` | App under test; steps say "the login page", not full URLs |
49
49
  | `storageState` | Playwright storage-state JSON so replays and the agent start authenticated (`npx playwright open --save-storage=.auth/state.json <url>`) |
50
+ | `sessions` | Per browser session (`As "buyer" ...`): `{ "buyer": { "storageState": ".auth/buyer.json" } }` starts that session signed in as its own person; others, and the default session, take `storageState` |
50
51
  | `actionTimeoutMs` | Budget for one action: Playwright's actionability wait and the deadline for a polled assertion (default 5000) |
51
52
  | `pollIntervalMs` | How often a polled assertion re-checks the page inside that budget (default 100, minimum 10) |
52
53
  | `screenshot` | Screenshot of the failing tab when a step fails: `viewport` (default), `full-page`, `off`. Shown in the report; paths in `latest.json` under `evidence`; files in `.saffron/artifacts/` (git-ignore it) |
@@ -58,8 +59,10 @@ your-project/
58
59
  | `snapshotMode` | `none` (default, ~60% fewer AI calls) or `full` for highly dynamic pages |
59
60
  | `setup` / `teardown` | shell command or list, run once before / after the run in the project root (seed and clean test data). Failed setup: exit 2, nothing runs. Teardown always runs. They receive `SAFFRON_BASE_URL` and `SAFFRON_ENV`. `--no-hooks` skips them; `hookTimeoutMs` (default 300000) bounds each |
60
61
  | `trace` | `off` (default), `on` or `retain-on-failure`: keep an execution trace per scenario; `saffron trace` opens it as a replay with the page inspectable at every step |
61
- | `browser` | `chromium` / `chrome` / `msedge` / `firefox` / `webkit`: replay runs anywhere; recording and healing need one of the first three. `chrome` and `msedge` must be installed on the machine |
62
- | `workers` | Parallel replay workers; agent work stays sequential |
62
+ | `browser` | `chromium` / `chrome` / `msedge` / `firefox` / `webkit`: replay runs anywhere; recording and healing need one of the first three. `chrome` and `msedge` must be installed on the machine. A list runs the suite in each into one report (browser matrix; Chromium-family first) |
63
+ | `device` / `viewport` | A Playwright device profile (`"iPhone 15"`) and/or a `"WIDTHxHEIGHT"` size for every scenario; a desktop recording replays under it or fails at the step that differs |
64
+ | `workers` | Parallel replay workers |
65
+ | `agentWorkers` | Agent sessions (record, heal) that may run at once, each in its own browser (default 1; each is a paid session) |
63
66
  | `shardBy` | How `--shard` splits the suite: `"scenario"` (default) or `"file"` (each feature file on one machine) |
64
67
  | `provider` | Whose agent records and heals, on its login on this machine: `claude` (default), `codex` (ChatGPT plan), `antigravity` (Google AI Pro/Ultra via the `agy` CLI, experimental), `cursor`. Also `--provider` |
65
68
  | `model` | Model in the provider's own names (unset: the provider's default). A `--provider` run on another provider ignores it |
@@ -70,8 +73,8 @@ your-project/
70
73
 
71
74
  | Command | Purpose |
72
75
  |---|---|
73
- | `saffron run [paths] [--filter <tag expression>] [--no-agent] [--strict] [--browser b] [--workers n] [--shard i/n] [--shard-by scenario\|file] [--provider p] [--heal-model m] [--headed] [--trace mode]` | Run; cached replay, agent on misses/failures. `--filter` takes tags or an expression (`"@e2e and not @wip"`). `--shard 2/4` runs one part of the suite on one CI machine and writes `shard-2-of-4.json` instead of `latest.json`; `--trace retain-on-failure` keeps an execution trace of each scenario that is not green |
74
- | `saffron trace [scenario] [--port n] [--no-open] [--json] [--run startedAt]` | Open the execution replay of a scenario from the last run (scenario name, feature path or `feature:scenario`; default: every traced scenario in one replay, opening on a red one, else a yellow one); `--json` prints the parsed trace; `--run` opens it only while the last run is that one; `--run <CI run id>` replays a CI run's traces from `.saffron/ci/` instead, view only (no Accept or Reject) |
76
+ | `saffron run [paths] [--filter <tag expression>] [--no-agent] [--strict] [--browser b[,b...]] [--device name] [--viewport WxH] [--workers n] [--agent-workers n] [--shard i/n] [--shard-by scenario\|file] [--provider p] [--heal-model m] [--headed] [--trace mode]` | Run; cached replay, agent on misses/failures. `--filter` takes tags or an expression (`"@e2e and not @wip"`). `--shard 2/4` runs one part of the suite on one CI machine and writes `shard-2-of-4.json` instead of `latest.json`; `--trace retain-on-failure` keeps an execution trace of each scenario that is not green |
77
+ | `saffron trace [scenario] [--port n] [--no-open] [--json] [--run startedAt]` | Open the execution replay of a scenario from the last run (scenario name, feature path or `feature:scenario`, `feature:scenario@browser` after a browser matrix; default: every traced scenario in one replay, opening on a red one, else a yellow one); `--json` prints the parsed trace; `--run` opens it only while the last run is that one; `--run <CI run id>` replays a CI run's traces from `.saffron/ci/` instead, view only (no Accept or Reject) |
75
78
  | `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote proposals (`--all` skips ones that are UNVERIFIED or not checked unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`; a named file is accepted as before); `--with-feature-edit` rewrites adapted steps in the feature file; `--propagate` applies a heal's locator fix to every cache using that locator |
76
79
  | `saffron reject [files... \| --all]` | Discard proposals |
77
80
  | `saffron prune [--yes] [--check] [--json]` | List recordings no scenario owns any more; `--yes` deletes them, `--check` exits 1 for CI |
@@ -81,11 +84,11 @@ your-project/
81
84
  | `saffron login [provider]` | Check that a provider can record and heal here and list what is missing; `saffron login cursor` also runs Cursor's one-time sign-in, `saffron login antigravity` also allows Saffron's tools in agy |
82
85
  | `saffron mcp` / `saffron lsp` | MCP tools (`search_steps`, `list_step_sets`, `project_status`) for AI assistants / language server for editors |
83
86
  | `saffron init [--examples] [--agents list]` | Install this skill into the project's agent directories, register MCP, add an AGENTS.md block, scaffold config; `--examples` adds the Saucedemo example suite |
84
- | `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs, 1 GB, fixed), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download |
87
+ | `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs, 1 GB, fixed), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
85
88
  | `saffron dashboard [--stdout] [--no-open]` | The suite's quality from the run history (failing and for how long, flaky, repeatedly healed, never run, slowest, trends, per tag); writes and opens `.saffron/reports/dashboard.html`, `--stdout` prints it |
86
- | `saffron export [-o file] [--branch name [--base branch]]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
89
+ | `saffron export [-o file] [--branch name [--base branch]] [--report-only]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). `--report-only` writes a summary instead (report and screenshots only, `saffron-summary.zip`): upload it as a `saffron-summary*` artifact on every run so any run's results can be viewed; a run that left only summaries has nothing to import. In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
87
90
  | `saffron runs [--branch b] [--limit n] [--provider github\|azure] [--json]` | The branch's CI runs, with how many of each run's proposals are pending here; exit 2 when `gh` or `az` is missing or signed out |
88
- | `saffron import <bundles...> \| --run <id> [--allow-fork] [--overwrite] [--allow-unknown-repository] [--json]` | Bring a CI run's bundle into this checkout: its proposals (each checked against the files here), report and traces. `--run` refuses a run a fork's pull request started unless `--allow-fork`. Then `saffron diff`, and `saffron accept <file>` (or `--all --include-imported`). Exit 1 when a proposal was not imported, 2 when the bundle was refused |
91
+ | `saffron import <bundles...> \| --run <id> [--allow-fork] [--overwrite] [--allow-unknown-repository] [--json]` | Bring a CI run's bundle (a file, a folder holding `saffron-bundle*.zip` or `saffron-summary*.zip`, or an http(s) URL) into this checkout: its proposals (each checked against the files here), report and traces. `--run` refuses a run a fork's pull request started unless `--allow-fork`. Then `saffron diff`, and `saffron accept <file>` (or `--all --include-imported`). Exit 1 when a proposal was not imported, 2 when the bundle was refused |
89
92
 
90
93
  Exit codes: 0 green/yellow, 1 red (or yellow with `--strict`), 2 usage /
91
94
  preflight (e.g. a missing `{env:VAR}`) or any expected failure, printed as
@@ -1,6 +1,8 @@
1
1
  # Saffron syntax cheat sheet
2
2
 
3
- Everything valid Gherkin is valid Saffron. `.saffron` files add `StepSet:`.
3
+ A `.feature` file runs as it is. `.saffron` files go further: `StepSet:`,
4
+ several people per scenario (`As "name"`), `{data:...}` outlines and
5
+ everything below.
4
6
 
5
7
  ## Steps
6
8
 
@@ -143,6 +145,11 @@ is 9 digits.
143
145
  - Say the intent: "select a check-in date 1 day from today" → recorded
144
146
  as `{date+1}`; stays valid every day. Formats: `{date}`, `{date-3}`,
145
147
  `{date+1:DD.MM.YYYY}`.
148
+ - A date written out (step text, table, doc string, `{data:...}`,
149
+ `{env:...}`) is entered as written on every run. In an Outline, each
150
+ Examples row's `{date+N}` cell, in any format, gives that row its own day.
151
+ - Text assertions read only what the page shows: hidden text, hidden
152
+ options and hidden parts of an element never satisfy them.
146
153
  - Capture and compare displayed values instead of literals:
147
154
 
148
155
  ```gherkin
@@ -192,6 +199,29 @@ Recorded as `switchTab` / `closeTab` / `newTab` with 0-based tab indexes
192
199
  in creation order; replay repeats them in order. Say which tab in the
193
200
  step ("the first tab", "the preview tab") so the recording is unambiguous.
194
201
 
202
+ ## Several people: `As "name"` (browser sessions)
203
+
204
+ ```gherkin
205
+ Given As "seller" I am on the orders page
206
+ When As "buyer" I order the blue lamp # the buyer's own browser
207
+ Then As "seller" I see an order for the blue lamp
208
+ And the order is marked new # still the seller
209
+ ```
210
+
211
+ A step starting `As "name"` runs in that person's browser session: a
212
+ browser context with its own cookies, storage, login and tabs. The
213
+ session carries over to the following steps until another `As "..."`;
214
+ steps before the first one run in the default session, and `As "default"`
215
+ goes back to it. Names are case-sensitive. Each session keeps
216
+ its page while another one acts. Give a session its own login with
217
+ `"sessions": { "buyer": { "storageState": ".auth/buyer.json" } }` in
218
+ `saffron.config.json`; a session without an entry starts from the
219
+ top-level `storageState`, and the run warns when two such people would
220
+ start as the same user. Keep the `As "..."` when rewording a step: it
221
+ decides where that step and the ones after it run. A recording is reused
222
+ only in its own session (named or carried over); a session taken from an
223
+ Examples column (`As "<who>"`) is never reused across rows.
224
+
195
225
  ## StepSets (`.saffron` only)
196
226
 
197
227
  ```gherkin
@@ -238,3 +268,74 @@ Scenario: ...
238
268
  @flaky"`). A comma, `|`, a space or a repeated `--filter` between plain
239
269
  tags means `or`; once `and`, `not` or a parenthesis appears, join every
240
270
  tag with `and` or `or`. The `@` may be left out.
271
+
272
+ ## A complete file
273
+
274
+ How the pieces fit in one project. The shared sign-in lives in a
275
+ sets-only library file:
276
+
277
+ ```gherkin
278
+ # features/shared.steps.saffron
279
+ Feature: Shared steps
280
+
281
+ StepSet: Sign in as admin
282
+ Given I am on the sign-in page
283
+ When I enter "{data:users.admin.email}" in the email field
284
+ And I enter "{env:ADMIN_PASSWORD}" in the password field
285
+ And I click the "Sign in" button
286
+ Then I should see the dashboard
287
+ ```
288
+
289
+ ```gherkin
290
+ # features/orders.saffron
291
+ Feature: Orders
292
+
293
+ Background:
294
+ Given I open the application
295
+ And I accept cookies if prompted
296
+
297
+ @smoke @orders
298
+ Scenario: Admin cancels a new order
299
+ StepSet Sign in as admin
300
+ When I open the orders page
301
+ And I filter the orders by status "New"
302
+ And I record the first order number as "order"
303
+ And I cancel the first order and confirm the dialog
304
+ Then "order" should be shown with status "Cancelled"
305
+ And I should not see an error banner
306
+
307
+ @orders
308
+ Scenario: Admin adds a note to an order
309
+ StepSet Sign in as admin
310
+ When I open the first order
311
+ And I add the note
312
+ """
313
+ Customer asked for delivery after 5pm.
314
+ """
315
+ Then the saved note should be shown
316
+
317
+ @smoke
318
+ Scenario Outline: Each role sees its own menu
319
+ Given I sign in as a <role> user
320
+ Then I should see the "<menu>" menu
321
+
322
+ Examples: {data:roles}
323
+ ```
324
+
325
+ `data/users.json` holds `{"admin": {"email": "admin@shop.test"}}` and
326
+ `data/roles.csv` has a `role,menu` header with one row per role;
327
+ `ADMIN_PASSWORD` comes from the environment or a git-ignored `.env`.
328
+ What each part buys:
329
+
330
+ - **Background** holds only what every scenario needs; anything that
331
+ differs per scenario belongs in the scenario.
332
+ - **The StepSet** signs in identically everywhere, so it records once
333
+ and every caller replays it.
334
+ - **Captured values** (`record ... as "order"`) instead of a literal
335
+ order number, which changes every run.
336
+ - **The doc string** records as `<docstring>`: reword the note and
337
+ replay stays free.
338
+ - **The outline** records once on the first row; every other row, and
339
+ every row added to `roles.csv` later, replays at zero tokens.
340
+ - **Each scenario ends in its `Then` block**, the strict verdict that is
341
+ never healed.