saffron-ai 0.9.8 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +485 -0
- package/README.md +4 -4
- package/dist-pkg/cli.js +371 -240
- package/package.json +4 -3
- package/skills/saffron/SKILL.md +183 -87
- package/skills/saffron/references/config.md +10 -7
- package/skills/saffron/references/syntax.md +102 -1
package/package.json
CHANGED
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "saffron-ai",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "1.0.0",
|
|
4
4
|
"description": "AI-fallback test runner for end-to-end tests written in plain language: zero-token cached replay, runtime AI healing, honest reports",
|
|
5
5
|
"scripts": {
|
|
6
6
|
"test": "vitest run",
|
|
7
7
|
"build": "tsc",
|
|
8
8
|
"dev": "tsc --watch",
|
|
9
9
|
"e2e": "node scripts/e2e.mjs",
|
|
10
|
-
"prepublishOnly": "npm run build && npm test
|
|
10
|
+
"prepublishOnly": "npm run build && npm test",
|
|
11
|
+
"prepack": "npm run bundle",
|
|
11
12
|
"docs": "node scripts/build-docs.mjs && node scripts/build-site-docs.mjs && node scripts/build-claude-plugin.mjs && node scripts/build-vscode-keywords.mjs",
|
|
12
13
|
"bundle": "esbuild src/cli/index.ts --bundle --platform=node --format=esm --packages=external --minify --legal-comments=none --outfile=dist-pkg/cli.js",
|
|
13
14
|
"plugin": "node scripts/build-claude-plugin.mjs"
|
|
@@ -65,7 +66,7 @@
|
|
|
65
66
|
"url": "https://github.com/s-chathuranga-j/saffron-community/issues"
|
|
66
67
|
},
|
|
67
68
|
"engines": {
|
|
68
|
-
"node": ">=
|
|
69
|
+
"node": ">=22.12"
|
|
69
70
|
},
|
|
70
71
|
"files": [
|
|
71
72
|
"dist-pkg",
|
package/skills/saffron/SKILL.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: saffron
|
|
3
|
-
description:
|
|
3
|
+
description: Create, translate, or edit Saffron UI test scenarios in .saffron or .feature files, including validating their recordings and replay. Use when asked to author or fix these test files in a Saffron project. Do not activate for ordinary application development, generic Cucumber projects, or requests only to run existing tests, inspect reports, review proposals, or configure Saffron.
|
|
4
4
|
license: Saffron Free Use License v1.0 (see the saffron-ai package LICENSE)
|
|
5
5
|
metadata:
|
|
6
6
|
author: saffron-ai
|
|
@@ -9,8 +9,19 @@ metadata:
|
|
|
9
9
|
|
|
10
10
|
# Saffron
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
|
|
12
|
+
Apply this workflow only when asked to create, translate, or edit Saffron
|
|
13
|
+
scenarios in `.saffron` or `.feature` files. A Saffron mention or an
|
|
14
|
+
existing test file in the project does not activate authoring work.
|
|
15
|
+
The run, review, and configuration commands below support that authoring
|
|
16
|
+
task; they are not instructions to change unrelated tests or settings.
|
|
17
|
+
Respect draft-only or no-execution requests and report the remaining
|
|
18
|
+
validation explicitly.
|
|
19
|
+
|
|
20
|
+
Saffron runs end-to-end tests written in plain language, in a real
|
|
21
|
+
browser. A `.saffron` file reads like Given/When/Then and goes further:
|
|
22
|
+
StepSets, several people in one scenario (`As "buyer"`), data tokens and
|
|
23
|
+
data-driven outlines; a plain `.feature` file runs as it is. There are
|
|
24
|
+
**no step definitions and no glue code**: on the first run an AI agent performs
|
|
14
25
|
each step and records what it did into a JSON cache; every later run
|
|
15
26
|
is replayed by Saffron in a real browser: zero AI calls, zero tokens, at
|
|
16
27
|
the speed of the browser and the app. (The cache is Saffron's own JSON, run
|
|
@@ -36,9 +47,11 @@ npx saffron steps "login" # search
|
|
|
36
47
|
npx saffron steps --json # machine-readable
|
|
37
48
|
```
|
|
38
49
|
|
|
39
|
-
Prefer a `●` recorded wording verbatim. Quoted values may differ
|
|
50
|
+
Prefer a `●` recorded wording verbatim. Quoted values may differ
|
|
40
51
|
(`I enter "admin"` seeds from `I enter "bob"`), so parameterize with
|
|
41
|
-
quotes rather than inventing new phrasings.
|
|
52
|
+
quotes rather than inventing new phrasings. A recording that still
|
|
53
|
+
names the old value elsewhere (a `delete-bob` test id, a `#row-bob`
|
|
54
|
+
selector) is not reused for the new one: that step records afresh. `saffron init` registers the
|
|
42
55
|
`saffron` MCP server with Claude Code, Cursor and VS Code; its
|
|
43
56
|
`search_steps` / `list_step_sets` / `project_status` tools give the same
|
|
44
57
|
answers without a shell.
|
|
@@ -55,52 +68,149 @@ cached (editing their text re-records them), which proposals await
|
|
|
55
68
|
review, and whether the vocabulary has divergent steps or duplicate
|
|
56
69
|
wordings to converge before adding more.
|
|
57
70
|
|
|
58
|
-
## 2. `Then` steps are
|
|
71
|
+
## 2. `Then` steps are the verdict
|
|
59
72
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
73
|
+
Final `Then` steps must check the state reached by the actions. Never
|
|
74
|
+
change guests, rooms, filters, or other application state inside an
|
|
75
|
+
assertion to manufacture its expected result. Mid-scenario assertions
|
|
76
|
+
may adapt only under the project's explicit policy; final assertions
|
|
77
|
+
remain strict. State exactly what must be true using stable text or
|
|
78
|
+
named captures for dynamic values (prices, dates, counters). A `Then`
|
|
63
79
|
that records no check at all fails on replay, so every assertion step must
|
|
64
80
|
end up with something recorded against it. Put
|
|
65
81
|
the final assertions **last**; the trailing block of `Then` steps is
|
|
66
|
-
strict under every policy. Use `Given`/`When` for actions.
|
|
82
|
+
strict under every policy. Use `Given`/`When` for actions. A text check
|
|
83
|
+
passes only on text the page shows: hidden elements, hidden parts of an
|
|
84
|
+
element and hidden dropdown options never count, so "should be shown"
|
|
85
|
+
means visible.
|
|
67
86
|
|
|
68
87
|
## 3. Never bake volatile or secret values into files
|
|
69
88
|
|
|
70
89
|
- Secrets: `{env:VAR}`, as in `When I enter "{env:ADMIN_PASSWORD}" in the password field`.
|
|
71
90
|
- Shared test data (accounts, names, enum values): put it in `data/*.json` and write `{data:users.admin.email}` (file, then dotted path; one value, not a list). The recording stores the token, so editing the file changes the next replay at zero tokens. Never for secrets: data files are committed. To check a whole list (an enum), point a Then step at it: `Then the status filter should list every {data:enums.OrderStatus}`; replay iterates the file's list, so adding a value needs no re-record. `Examples: {data:roles}` reads outline rows from `data/roles.csv` or a JSON list (`.saffron` only). A value that must be new on every run (an email for a sign-up) is `{unique:name}`: fresh per run, the same within it.
|
|
72
91
|
- State the application must be in before the suite (seeded users, a reset database): do not script it through the UI in every scenario. The project's `saffron.config.json` takes `"setup"` and `"teardown"` shell commands, run once around `saffron run` (teardown always runs). Propose a seed script there; keep scenario steps for the behaviour under test.
|
|
73
|
-
- Dates: write intent
|
|
92
|
+
- Dates: write intent when the day should move ("1 day from today" records as `{date+1}`); a date you write out (`I enter 2026-10-10`, in a table, a data file or `{env:...}`) is entered as written on every run. An Examples cell such as `{date+2:DD.MM.YYYY}` gives each row its own day.
|
|
74
93
|
- Dynamic display values: capture and compare, e.g. `I record the total as "first"` … `"first" should differ from the displayed total`.
|
|
75
94
|
|
|
76
|
-
##
|
|
95
|
+
## Authoring workflow
|
|
96
|
+
|
|
97
|
+
1. **Survey once.** Read the relevant scenarios and project config, then
|
|
98
|
+
`saffron status --json` (or `project_status`). Search the vocabulary
|
|
99
|
+
with `saffron steps "<words>"` / `search_steps` and check `list_step_sets`.
|
|
100
|
+
Reuse recorded wording only when it expresses the same behavior.
|
|
101
|
+
2. **Inspect the behavior.** For a new or uncertain flow, use the available
|
|
102
|
+
browser tools to walk the relevant actions and observe their results.
|
|
103
|
+
Check the actual labels, initial state, comparison target, and loading
|
|
104
|
+
signal. Exercise each transition you will assert; seeing the controls
|
|
105
|
+
alone is insufficient. Existing tests and source code help locate the
|
|
106
|
+
flow, but do not establish its current UI behavior. Reuse recent,
|
|
107
|
+
relevant browser evidence instead of exploring the whole application.
|
|
108
|
+
If access or test data is missing, identify the gap and produce a draft
|
|
109
|
+
with explicit assumptions; do not invent observations.
|
|
110
|
+
3. **Design one behavior per scenario.** When translating Cypress or other
|
|
111
|
+
tests, preserve the intended coverage, reconsider assumptions, and
|
|
112
|
+
split independent changes. Compare the same target immediately before
|
|
113
|
+
and after the change under test. Do not assume every intermediate value
|
|
114
|
+
must differ from every earlier value. Keep required setup and waits;
|
|
115
|
+
there is no fixed step-count limit.
|
|
116
|
+
4. **Write to the contract below.** Add to the feature for that area. In
|
|
117
|
+
`.saffron`, reuse StepSets for repeated flows; `.feature` uses standard
|
|
118
|
+
Gherkin without StepSets or file-backed Examples. Resolve syntax and
|
|
119
|
+
data diagnostics and check existing wording before recording.
|
|
120
|
+
5. **Record the affected scenarios.** Select each new or changed scenario
|
|
121
|
+
by line or name, for example `saffron run features/x.saffron:<line>`.
|
|
122
|
+
If a shared StepSet changes, include its affected callers. Avoid a
|
|
123
|
+
whole-suite recording or disabling reuse by default. Re-record a faulty
|
|
124
|
+
recording only after diagnosing its failed step and proof evidence.
|
|
125
|
+
6. **Review what was recorded.** Use `saffron diff` and the report/trace.
|
|
126
|
+
Check that captures and comparisons use the same names and intended
|
|
127
|
+
elements, waits observe the real signal, and final assertions contain
|
|
128
|
+
no state-changing actions or replacement baselines. A `verified ✓`
|
|
129
|
+
proposal proves replay passed; it does not prove the recording faithfully
|
|
130
|
+
tests the written requirement. Investigate `UNVERIFIED` proposals; do
|
|
131
|
+
not accept them to finish the task or weaken the expected behavior.
|
|
132
|
+
7. **Prove completion.** Accept only the reviewed, verified proposals for
|
|
133
|
+
this task, then run the affected scenarios with `--no-agent`. Complete
|
|
134
|
+
means those scenarios pass with zero AI calls and zero tokens, with
|
|
135
|
+
their intended assertions preserved. Report exactly what passed and
|
|
136
|
+
any scenarios or prerequisites not checked. If acceptance is reserved
|
|
137
|
+
for someone else or execution is unavailable/excluded, deliver the
|
|
138
|
+
draft or pending proposals and name the unfinished validation. Keep
|
|
139
|
+
feature files, shared data, and accepted caches together for review;
|
|
140
|
+
commit only when the task or project workflow calls for it.
|
|
141
|
+
|
|
142
|
+
## Writing steps that record reliably
|
|
143
|
+
|
|
144
|
+
- **One clear intent per step.** Use visible labels, roles, and names:
|
|
145
|
+
`the "Save" button`, `the email field`. Keep CSS selectors, XPath, test
|
|
146
|
+
ids, and component names out of scenario prose. Granularity means an
|
|
147
|
+
independently understandable action or check, not necessarily one click.
|
|
148
|
+
- **Name captured values explicitly.** Write `I record the displayed price
|
|
149
|
+
for the selected date as "beforePrice"`, then `the displayed price for
|
|
150
|
+
the selected date should differ from "beforePrice"`. Reuse the exact
|
|
151
|
+
name; avoid `the price I noted` or `the previous value`. Record a baseline
|
|
152
|
+
before the action, and never recreate it inside the final assertion.
|
|
153
|
+
- **Keep the target and value comparable.** Compare the same date, item,
|
|
154
|
+
or row across the change. If testing an amount change, capture amount-only
|
|
155
|
+
text, not a whole button label containing a date, currency, or guest count.
|
|
156
|
+
Saffron's capture stores text; calling it "numeric" does not add numeric
|
|
157
|
+
parsing or arithmetic. Do not request unsupported calculations as though
|
|
158
|
+
they were available replay operations.
|
|
159
|
+
- **Wait for an observed result.** State the relevant visible state or API
|
|
160
|
+
response instead of `finished loading` without a signal or a fixed sleep.
|
|
161
|
+
A currency option called `GBP` may display `£` on prices. Use the observed
|
|
162
|
+
display value for the price wait; a changed selector alone does not prove
|
|
163
|
+
prices refreshed. Do not wait for the expected assertion outcome merely
|
|
164
|
+
to make it pass; the final check must still establish the requirement.
|
|
165
|
+
- **Keep test inputs explicit.** Use the data, secret, and date rules above.
|
|
166
|
+
Establish the starting guests/rooms or other relevant state. Name required
|
|
167
|
+
seeded data instead of silently assuming it exists. Conditional actions
|
|
168
|
+
such as `accept cookies if prompted` record a no-op when absent; that is
|
|
169
|
+
not a promise that replay will discover every future branch.
|
|
170
|
+
|
|
171
|
+
For example, after selecting a date and observing its price:
|
|
77
172
|
|
|
78
173
|
```gherkin
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
When I add "Blue Widget" to the cart
|
|
88
|
-
And I place the order and wait for the order API to return 201
|
|
89
|
-
Then I should see "Order placed"
|
|
90
|
-
And the order request should have returned 201
|
|
174
|
+
Scenario: Changing two adults to one updates the selected date's price
|
|
175
|
+
Given the rooms and guests button shows "1 room, 2 adults"
|
|
176
|
+
When I record the displayed calendar price for the selected date as "beforePrice"
|
|
177
|
+
And I open the rooms and guests panel
|
|
178
|
+
And I reduce the adults in room 1 from 2 to 1
|
|
179
|
+
And I wait for the calendar price refresh indicator to disappear
|
|
180
|
+
Then the rooms and guests button should read "1 room, 1 adult"
|
|
181
|
+
And the displayed calendar price for the selected date should differ from "beforePrice"
|
|
91
182
|
```
|
|
92
183
|
|
|
93
|
-
|
|
184
|
+
Use this shape only where the app actually has that refresh indicator and
|
|
185
|
+
pricing rule. Setup must select a date with suitable test availability.
|
|
186
|
+
An independent child or second-room price change belongs in its own
|
|
187
|
+
scenario with its own baseline. For currency changes, distinguish the
|
|
188
|
+
selector's code from the observed display symbol and capture amount-only
|
|
189
|
+
text. See [syntax.md](references/syntax.md#a-complete-file) for a full file.
|
|
94
190
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
191
|
+
`saffron author` drafts from supplied requirements and vocabulary without
|
|
192
|
+
browser inspection. Include observed labels, waits, prerequisites, and
|
|
193
|
+
expected behavior in its input. Its output must still go through steps
|
|
194
|
+
5 to 7 above before it is described as a working test.
|
|
195
|
+
|
|
196
|
+
## Several people in one scenario: `As "name"`
|
|
197
|
+
|
|
198
|
+
```gherkin
|
|
199
|
+
Given As "seller" I am on the orders page
|
|
200
|
+
When As "buyer" I order the blue lamp # the buyer's own browser
|
|
201
|
+
Then As "seller" I see an order for the blue lamp
|
|
202
|
+
And the order is marked new # still the seller
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Each name is its own browser session (cookies, login, tabs). It carries
|
|
206
|
+
over to the next steps until another `As "..."`; `As "default"` goes
|
|
207
|
+
back. Give each person a login with `"sessions": { "buyer": {
|
|
208
|
+
"storageState": ".auth/buyer.json" } }`; a person without an entry
|
|
209
|
+
starts from the top-level `storageState` (two such people start as the
|
|
210
|
+
same user, and the run warns). Keep a step's `As "..."` when rewording
|
|
211
|
+
it, and never copy one person's prefix onto another's step: a recording
|
|
212
|
+
is reused only within its own session, so the buyer's typed address
|
|
213
|
+
never seeds the seller's step.
|
|
104
214
|
|
|
105
215
|
## Repeated step groups → StepSets (`.saffron` files only)
|
|
106
216
|
|
|
@@ -128,60 +238,45 @@ Scenario: Checkout happy path
|
|
|
128
238
|
`Feature:` header; it yields no runnable scenarios).
|
|
129
239
|
- Start a set with a guard step, end it with an exit assertion.
|
|
130
240
|
|
|
131
|
-
##
|
|
241
|
+
## Commands for completing an authored test
|
|
132
242
|
|
|
133
243
|
```bash
|
|
134
|
-
npx saffron run
|
|
135
|
-
npx saffron
|
|
136
|
-
npx saffron
|
|
137
|
-
npx saffron
|
|
138
|
-
npx saffron run
|
|
139
|
-
npx saffron status
|
|
140
|
-
npx saffron diff # what a proposal changes, action by action: the evidence
|
|
141
|
-
npx saffron accept # list pending proposals with narratives
|
|
142
|
-
npx saffron accept --all # promote verified proposals to caches
|
|
143
|
-
npx saffron accept <file> [<file>...] # promote chosen ones
|
|
144
|
-
npx saffron reject <file> [<file>...] # discard chosen ones; the agent retries next run
|
|
145
|
-
npx saffron report # open the HTML report
|
|
146
|
-
npx saffron dashboard # suite quality: failing, flaky, never run, slowest, per tag
|
|
147
|
-
npx saffron run --rerecord --filter @t # a recording is wrong: record it fresh
|
|
148
|
-
npx saffron trace "<scenario>" # a red or healed scenario, step by step, with the page at every action ("trace" on in config)
|
|
149
|
-
npx saffron runs # this branch's CI runs; a heal made there: saffron import --run <id>
|
|
150
|
-
npx saffron report --run <id> # a CI run's results without importing (kept in .saffron/ci/)
|
|
244
|
+
npx saffron run features/login.saffron:12 # record/replay one affected scenario
|
|
245
|
+
npx saffron diff # inspect pending recordings
|
|
246
|
+
npx saffron trace "<scenario>" # inspect a failed or adapted step (trace enabled)
|
|
247
|
+
npx saffron accept <proposal-file> # accept a reviewed, verified proposal
|
|
248
|
+
npx saffron run features/login.saffron:12 --no-agent # prove cached replay
|
|
249
|
+
npx saffron status --json # confirm affected scenario state
|
|
151
250
|
```
|
|
152
251
|
|
|
153
|
-
Results: **green** = cached pass · **yellow** = pending review
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
report and traces here, each checked against this checkout; then
|
|
182
|
-
`saffron diff`, `saffron accept` and commit `.saffron/cache/` as usual.
|
|
183
|
-
With `saffron export --branch <name>` CI pushes the proposals as a branch
|
|
184
|
-
with a draft pull request instead: switch to it, accept, commit, push.
|
|
252
|
+
Results: **green** = cached pass · **yellow** = pending review · **red** =
|
|
253
|
+
failed. `verified ✓` means the proposal's zero-AI proof replay passed;
|
|
254
|
+
review its meaning before accepting. `UNVERIFIED ✗` needs investigation.
|
|
255
|
+
A STALE proposal belongs to an older scenario or recording: run the
|
|
256
|
+
scenario again for a fresh one. Keep accepted `.saffron/cache/` files
|
|
257
|
+
with the test; ignore `.saffron/reports/`.
|
|
258
|
+
|
|
259
|
+
For outlines, check every affected Examples row with `--no-agent`, not
|
|
260
|
+
only the row used to record. Quote the row's expected value in `Then`
|
|
261
|
+
(e.g. `Then I should see "<message>"`) so the recording keeps the parameter.
|
|
262
|
+
One recording serves every row, and it cannot tell whether a row value
|
|
263
|
+
glued inside other text belongs to the row (an email built from `<user>`,
|
|
264
|
+
`INV20261013`, `orders-US`) or a field lists every row's value. Write
|
|
265
|
+
`<column>` in every step whose text or typed value depends on the row;
|
|
266
|
+
otherwise the other rows fail red naming the column, and some cases
|
|
267
|
+
(dates without a year, two-letter values, blank cells) are not caught at
|
|
268
|
+
all. Keep a check that lists all rows' values out of the Outline.
|
|
269
|
+
Tests live in the folder named by `features` in `saffron.config.json`
|
|
270
|
+
(`features/` by default).
|
|
271
|
+
|
|
272
|
+
**Never hand-edit generated `.saffron/cache/` or `.saffron/proposals/` to
|
|
273
|
+
make a test pass.** Change incorrect scenario text and record again, or
|
|
274
|
+
use `saffron run <scenario-selector> --rerecord` when the recording is wrong.
|
|
275
|
+
Inspect failures before another paid recording; report an application,
|
|
276
|
+
fixture, or unsupported-operation blocker instead of repeatedly retrying
|
|
277
|
+
or weakening assertions. For CI artifacts, browser matrices, provider
|
|
278
|
+
setup, and other configuration needed by the requested test, read
|
|
279
|
+
[Configuration & CI](references/config.md).
|
|
185
280
|
|
|
186
281
|
## Anti-patterns that cost money or hide bugs
|
|
187
282
|
|
|
@@ -189,10 +284,10 @@ with a draft pull request instead: switch to it, accept, commit, push.
|
|
|
189
284
|
|---|---|
|
|
190
285
|
| Invent a synonym for an existing step | Reuse the `●` wording from `saffron steps` |
|
|
191
286
|
| Assert inside a `When` ("When I see the dashboard") | Act in `When`, assert in `Then` |
|
|
192
|
-
|
|
|
287
|
+
| Hard-code a changing price observed during authoring | Capture and compare for a change; use an exact price only when the requirement fixes it |
|
|
193
288
|
| Put a password literal in a step | `{env:VAR}` |
|
|
194
289
|
| Sleep/wait "for 3 seconds" | Wait for a UI signal or an API response |
|
|
195
|
-
|
|
|
290
|
+
| Several independent changes and all-pairs comparisons in one scenario | One behavior and its own baseline per scenario; reuse setup with StepSets in `.saffron` |
|
|
196
291
|
| Edit cache JSON to fix a test | Edit the scenario text and re-run, or `saffron run --rerecord` if the recording itself is wrong |
|
|
197
292
|
|
|
198
293
|
## References
|
|
@@ -200,4 +295,5 @@ with a draft pull request instead: switch to it, accept, commit, push.
|
|
|
200
295
|
- [Syntax cheat sheet](references/syntax.md): tables, doc strings,
|
|
201
296
|
outlines, secrets, dates, network waits, page furniture, StepSets.
|
|
202
297
|
- [Configuration & CI](references/config.md): `saffron.config.json`,
|
|
203
|
-
flags, `--strict`, cross-browser
|
|
298
|
+
flags, `--strict`, cross-browser and browser matrix, devices, workers
|
|
299
|
+
and agent workers, heal model, secrets setup.
|
|
@@ -47,6 +47,7 @@ your-project/
|
|
|
47
47
|
| `dataDir` | Folder of JSON files read by `{data:file.key}` tokens (default `data`). Committed; not for secrets |
|
|
48
48
|
| `baseURL` | App under test; steps say "the login page", not full URLs |
|
|
49
49
|
| `storageState` | Playwright storage-state JSON so replays and the agent start authenticated (`npx playwright open --save-storage=.auth/state.json <url>`) |
|
|
50
|
+
| `sessions` | Per browser session (`As "buyer" ...`): `{ "buyer": { "storageState": ".auth/buyer.json" } }` starts that session signed in as its own person; others, and the default session, take `storageState` |
|
|
50
51
|
| `actionTimeoutMs` | Budget for one action: Playwright's actionability wait and the deadline for a polled assertion (default 5000) |
|
|
51
52
|
| `pollIntervalMs` | How often a polled assertion re-checks the page inside that budget (default 100, minimum 10) |
|
|
52
53
|
| `screenshot` | Screenshot of the failing tab when a step fails: `viewport` (default), `full-page`, `off`. Shown in the report; paths in `latest.json` under `evidence`; files in `.saffron/artifacts/` (git-ignore it) |
|
|
@@ -58,8 +59,10 @@ your-project/
|
|
|
58
59
|
| `snapshotMode` | `none` (default, ~60% fewer AI calls) or `full` for highly dynamic pages |
|
|
59
60
|
| `setup` / `teardown` | shell command or list, run once before / after the run in the project root (seed and clean test data). Failed setup: exit 2, nothing runs. Teardown always runs. They receive `SAFFRON_BASE_URL` and `SAFFRON_ENV`. `--no-hooks` skips them; `hookTimeoutMs` (default 300000) bounds each |
|
|
60
61
|
| `trace` | `off` (default), `on` or `retain-on-failure`: keep an execution trace per scenario; `saffron trace` opens it as a replay with the page inspectable at every step |
|
|
61
|
-
| `browser` | `chromium` / `chrome` / `msedge` / `firefox` / `webkit`: replay runs anywhere; recording and healing need one of the first three. `chrome` and `msedge` must be installed on the machine |
|
|
62
|
-
| `
|
|
62
|
+
| `browser` | `chromium` / `chrome` / `msedge` / `firefox` / `webkit`: replay runs anywhere; recording and healing need one of the first three. `chrome` and `msedge` must be installed on the machine. A list runs the suite in each into one report (browser matrix; Chromium-family first) |
|
|
63
|
+
| `device` / `viewport` | A Playwright device profile (`"iPhone 15"`) and/or a `"WIDTHxHEIGHT"` size for every scenario; a desktop recording replays under it or fails at the step that differs |
|
|
64
|
+
| `workers` | Parallel replay workers |
|
|
65
|
+
| `agentWorkers` | Agent sessions (record, heal) that may run at once, each in its own browser (default 1; each is a paid session) |
|
|
63
66
|
| `shardBy` | How `--shard` splits the suite: `"scenario"` (default) or `"file"` (each feature file on one machine) |
|
|
64
67
|
| `provider` | Whose agent records and heals, on its login on this machine: `claude` (default), `codex` (ChatGPT plan), `antigravity` (Google AI Pro/Ultra via the `agy` CLI, experimental), `cursor`. Also `--provider` |
|
|
65
68
|
| `model` | Model in the provider's own names (unset: the provider's default). A `--provider` run on another provider ignores it |
|
|
@@ -70,8 +73,8 @@ your-project/
|
|
|
70
73
|
|
|
71
74
|
| Command | Purpose |
|
|
72
75
|
|---|---|
|
|
73
|
-
| `saffron run [paths] [--filter <tag expression>] [--no-agent] [--strict] [--browser b] [--workers n] [--shard i/n] [--shard-by scenario\|file] [--provider p] [--heal-model m] [--headed] [--trace mode]` | Run; cached replay, agent on misses/failures. `--filter` takes tags or an expression (`"@e2e and not @wip"`). `--shard 2/4` runs one part of the suite on one CI machine and writes `shard-2-of-4.json` instead of `latest.json`; `--trace retain-on-failure` keeps an execution trace of each scenario that is not green |
|
|
74
|
-
| `saffron trace [scenario] [--port n] [--no-open] [--json] [--run startedAt]` | Open the execution replay of a scenario from the last run (scenario name, feature path or `feature:scenario
|
|
76
|
+
| `saffron run [paths] [--filter <tag expression>] [--no-agent] [--strict] [--browser b[,b...]] [--device name] [--viewport WxH] [--workers n] [--agent-workers n] [--shard i/n] [--shard-by scenario\|file] [--provider p] [--heal-model m] [--headed] [--trace mode]` | Run; cached replay, agent on misses/failures. `--filter` takes tags or an expression (`"@e2e and not @wip"`). `--shard 2/4` runs one part of the suite on one CI machine and writes `shard-2-of-4.json` instead of `latest.json`; `--trace retain-on-failure` keeps an execution trace of each scenario that is not green |
|
|
77
|
+
| `saffron trace [scenario] [--port n] [--no-open] [--json] [--run startedAt]` | Open the execution replay of a scenario from the last run (scenario name, feature path or `feature:scenario`, `feature:scenario@browser` after a browser matrix; default: every traced scenario in one replay, opening on a red one, else a yellow one); `--json` prints the parsed trace; `--run` opens it only while the last run is that one; `--run <CI run id>` replays a CI run's traces from `.saffron/ci/` instead, view only (no Accept or Reject) |
|
|
75
78
|
| `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote proposals (`--all` skips ones that are UNVERIFIED or not checked unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`; a named file is accepted as before); `--with-feature-edit` rewrites adapted steps in the feature file; `--propagate` applies a heal's locator fix to every cache using that locator |
|
|
76
79
|
| `saffron reject [files... \| --all]` | Discard proposals |
|
|
77
80
|
| `saffron prune [--yes] [--check] [--json]` | List recordings no scenario owns any more; `--yes` deletes them, `--check` exits 1 for CI |
|
|
@@ -81,11 +84,11 @@ your-project/
|
|
|
81
84
|
| `saffron login [provider]` | Check that a provider can record and heal here and list what is missing; `saffron login cursor` also runs Cursor's one-time sign-in, `saffron login antigravity` also allows Saffron's tools in agy |
|
|
82
85
|
| `saffron mcp` / `saffron lsp` | MCP tools (`search_steps`, `list_step_sets`, `project_status`) for AI assistants / language server for editors |
|
|
83
86
|
| `saffron init [--examples] [--agents list]` | Install this skill into the project's agent directories, register MCP, add an AGENTS.md block, scaffold config; `--examples` adds the Saucedemo example suite |
|
|
84
|
-
| `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs, 1 GB, fixed), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download |
|
|
87
|
+
| `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs, 1 GB, fixed), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
|
|
85
88
|
| `saffron dashboard [--stdout] [--no-open]` | The suite's quality from the run history (failing and for how long, flaky, repeatedly healed, never run, slowest, trends, per tag); writes and opens `.saffron/reports/dashboard.html`, `--stdout` prints it |
|
|
86
|
-
| `saffron export [-o file] [--branch name [--base branch]]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
|
|
89
|
+
| `saffron export [-o file] [--branch name [--base branch]] [--report-only]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). `--report-only` writes a summary instead (report and screenshots only, `saffron-summary.zip`): upload it as a `saffron-summary*` artifact on every run so any run's results can be viewed; a run that left only summaries has nothing to import. In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
|
|
87
90
|
| `saffron runs [--branch b] [--limit n] [--provider github\|azure] [--json]` | The branch's CI runs, with how many of each run's proposals are pending here; exit 2 when `gh` or `az` is missing or signed out |
|
|
88
|
-
| `saffron import <bundles...> \| --run <id> [--allow-fork] [--overwrite] [--allow-unknown-repository] [--json]` | Bring a CI run's bundle into this checkout: its proposals (each checked against the files here), report and traces. `--run` refuses a run a fork's pull request started unless `--allow-fork`. Then `saffron diff`, and `saffron accept <file>` (or `--all --include-imported`). Exit 1 when a proposal was not imported, 2 when the bundle was refused |
|
|
91
|
+
| `saffron import <bundles...> \| --run <id> [--allow-fork] [--overwrite] [--allow-unknown-repository] [--json]` | Bring a CI run's bundle (a file, a folder holding `saffron-bundle*.zip` or `saffron-summary*.zip`, or an http(s) URL) into this checkout: its proposals (each checked against the files here), report and traces. `--run` refuses a run a fork's pull request started unless `--allow-fork`. Then `saffron diff`, and `saffron accept <file>` (or `--all --include-imported`). Exit 1 when a proposal was not imported, 2 when the bundle was refused |
|
|
89
92
|
|
|
90
93
|
Exit codes: 0 green/yellow, 1 red (or yellow with `--strict`), 2 usage /
|
|
91
94
|
preflight (e.g. a missing `{env:VAR}`) or any expected failure, printed as
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Saffron syntax cheat sheet
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A `.feature` file runs as it is. `.saffron` files go further: `StepSet:`,
|
|
4
|
+
several people per scenario (`As "name"`), `{data:...}` outlines and
|
|
5
|
+
everything below.
|
|
4
6
|
|
|
5
7
|
## Steps
|
|
6
8
|
|
|
@@ -143,6 +145,11 @@ is 9 digits.
|
|
|
143
145
|
- Say the intent: "select a check-in date 1 day from today" → recorded
|
|
144
146
|
as `{date+1}`; stays valid every day. Formats: `{date}`, `{date-3}`,
|
|
145
147
|
`{date+1:DD.MM.YYYY}`.
|
|
148
|
+
- A date written out (step text, table, doc string, `{data:...}`,
|
|
149
|
+
`{env:...}`) is entered as written on every run. In an Outline, each
|
|
150
|
+
Examples row's `{date+N}` cell, in any format, gives that row its own day.
|
|
151
|
+
- Text assertions read only what the page shows: hidden text, hidden
|
|
152
|
+
options and hidden parts of an element never satisfy them.
|
|
146
153
|
- Capture and compare displayed values instead of literals:
|
|
147
154
|
|
|
148
155
|
```gherkin
|
|
@@ -192,6 +199,29 @@ Recorded as `switchTab` / `closeTab` / `newTab` with 0-based tab indexes
|
|
|
192
199
|
in creation order; replay repeats them in order. Say which tab in the
|
|
193
200
|
step ("the first tab", "the preview tab") so the recording is unambiguous.
|
|
194
201
|
|
|
202
|
+
## Several people: `As "name"` (browser sessions)
|
|
203
|
+
|
|
204
|
+
```gherkin
|
|
205
|
+
Given As "seller" I am on the orders page
|
|
206
|
+
When As "buyer" I order the blue lamp # the buyer's own browser
|
|
207
|
+
Then As "seller" I see an order for the blue lamp
|
|
208
|
+
And the order is marked new # still the seller
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
A step starting `As "name"` runs in that person's browser session: a
|
|
212
|
+
browser context with its own cookies, storage, login and tabs. The
|
|
213
|
+
session carries over to the following steps until another `As "..."`;
|
|
214
|
+
steps before the first one run in the default session, and `As "default"`
|
|
215
|
+
goes back to it. Names are case-sensitive. Each session keeps
|
|
216
|
+
its page while another one acts. Give a session its own login with
|
|
217
|
+
`"sessions": { "buyer": { "storageState": ".auth/buyer.json" } }` in
|
|
218
|
+
`saffron.config.json`; a session without an entry starts from the
|
|
219
|
+
top-level `storageState`, and the run warns when two such people would
|
|
220
|
+
start as the same user. Keep the `As "..."` when rewording a step: it
|
|
221
|
+
decides where that step and the ones after it run. A recording is reused
|
|
222
|
+
only in its own session (named or carried over); a session taken from an
|
|
223
|
+
Examples column (`As "<who>"`) is never reused across rows.
|
|
224
|
+
|
|
195
225
|
## StepSets (`.saffron` only)
|
|
196
226
|
|
|
197
227
|
```gherkin
|
|
@@ -238,3 +268,74 @@ Scenario: ...
|
|
|
238
268
|
@flaky"`). A comma, `|`, a space or a repeated `--filter` between plain
|
|
239
269
|
tags means `or`; once `and`, `not` or a parenthesis appears, join every
|
|
240
270
|
tag with `and` or `or`. The `@` may be left out.
|
|
271
|
+
|
|
272
|
+
## A complete file
|
|
273
|
+
|
|
274
|
+
How the pieces fit in one project. The shared sign-in lives in a
|
|
275
|
+
sets-only library file:
|
|
276
|
+
|
|
277
|
+
```gherkin
|
|
278
|
+
# features/shared.steps.saffron
|
|
279
|
+
Feature: Shared steps
|
|
280
|
+
|
|
281
|
+
StepSet: Sign in as admin
|
|
282
|
+
Given I am on the sign-in page
|
|
283
|
+
When I enter "{data:users.admin.email}" in the email field
|
|
284
|
+
And I enter "{env:ADMIN_PASSWORD}" in the password field
|
|
285
|
+
And I click the "Sign in" button
|
|
286
|
+
Then I should see the dashboard
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
```gherkin
|
|
290
|
+
# features/orders.saffron
|
|
291
|
+
Feature: Orders
|
|
292
|
+
|
|
293
|
+
Background:
|
|
294
|
+
Given I open the application
|
|
295
|
+
And I accept cookies if prompted
|
|
296
|
+
|
|
297
|
+
@smoke @orders
|
|
298
|
+
Scenario: Admin cancels a new order
|
|
299
|
+
StepSet Sign in as admin
|
|
300
|
+
When I open the orders page
|
|
301
|
+
And I filter the orders by status "New"
|
|
302
|
+
And I record the first order number as "order"
|
|
303
|
+
And I cancel the first order and confirm the dialog
|
|
304
|
+
Then "order" should be shown with status "Cancelled"
|
|
305
|
+
And I should not see an error banner
|
|
306
|
+
|
|
307
|
+
@orders
|
|
308
|
+
Scenario: Admin adds a note to an order
|
|
309
|
+
StepSet Sign in as admin
|
|
310
|
+
When I open the first order
|
|
311
|
+
And I add the note
|
|
312
|
+
"""
|
|
313
|
+
Customer asked for delivery after 5pm.
|
|
314
|
+
"""
|
|
315
|
+
Then the saved note should be shown
|
|
316
|
+
|
|
317
|
+
@smoke
|
|
318
|
+
Scenario Outline: Each role sees its own menu
|
|
319
|
+
Given I sign in as a <role> user
|
|
320
|
+
Then I should see the "<menu>" menu
|
|
321
|
+
|
|
322
|
+
Examples: {data:roles}
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
`data/users.json` holds `{"admin": {"email": "admin@shop.test"}}` and
|
|
326
|
+
`data/roles.csv` has a `role,menu` header with one row per role;
|
|
327
|
+
`ADMIN_PASSWORD` comes from the environment or a git-ignored `.env`.
|
|
328
|
+
What each part buys:
|
|
329
|
+
|
|
330
|
+
- **Background** holds only what every scenario needs; anything that
|
|
331
|
+
differs per scenario belongs in the scenario.
|
|
332
|
+
- **The StepSet** signs in identically everywhere, so it records once
|
|
333
|
+
and every caller replays it.
|
|
334
|
+
- **Captured values** (`record ... as "order"`) instead of a literal
|
|
335
|
+
order number, which changes every run.
|
|
336
|
+
- **The doc string** records as `<docstring>`: reword the note and
|
|
337
|
+
replay stays free.
|
|
338
|
+
- **The outline** records once on the first row; every other row, and
|
|
339
|
+
every row added to `roles.csv` later, replays at zero tokens.
|
|
340
|
+
- **Each scenario ends in its `Then` block**, the strict verdict that is
|
|
341
|
+
never healed.
|