saffron-ai 0.9.7 → 0.9.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +155 -0
- package/README.md +2 -3
- package/dist-pkg/cli.js +200 -196
- package/package.json +2 -2
- package/skills/saffron/SKILL.md +2 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,161 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
|
|
|
5
5
|
JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
|
|
6
6
|
release heading names its main feature: `## <version>: <title>`.
|
|
7
7
|
|
|
8
|
+
## 0.9.8: Run one scenario, from the editor too
|
|
9
|
+
|
|
10
|
+
Released 2026-10-06.
|
|
11
|
+
|
|
12
|
+
Behaviour changes
|
|
13
|
+
|
|
14
|
+
- **Tags now apply as in Cucumber**: a Feature's and a Rule's tags to
|
|
15
|
+
every scenario in it, an Examples block's to its rows. A CI filter can
|
|
16
|
+
select differently than before: `not @ignore` with `@ignore` on a
|
|
17
|
+
Feature now skips that feature, and `--filter @smoke` with a
|
|
18
|
+
Feature-level `@smoke` now takes all of its scenarios. Check your
|
|
19
|
+
`--filter` expressions when you upgrade. The dashboard's results per tag
|
|
20
|
+
and the IDEs' Tags tabs count the inherited tags too.
|
|
21
|
+
|
|
22
|
+
Runner
|
|
23
|
+
|
|
24
|
+
- **Run one scenario by its line or its name.** `saffron run
|
|
25
|
+
features/login.saffron:12` runs the scenario that line is in,
|
|
26
|
+
Cucumber-style: its Scenario line, a tag above it or any line inside it.
|
|
27
|
+
An Examples row runs that row alone, any other line of an outline runs
|
|
28
|
+
every row, and `:12:30` names two. `features/login.saffron:Successful
|
|
29
|
+
login` names it instead, which stays true while the file is edited,
|
|
30
|
+
and `:Roles (example 2)` one Examples row, as `saffron trace` lists it. A
|
|
31
|
+
line no scenario owns (the feature's header, a Background, a StepSet, a
|
|
32
|
+
Rule's header, past the end), a name no scenario has, or an outline
|
|
33
|
+
whose Examples have no rows yet stops the run with exit 2 and says
|
|
34
|
+
what is there. Works with `--filter`, `--shard` and `--rerecord`.
|
|
35
|
+
- **Tags on a Feature, a Rule or an Examples block apply to what they
|
|
36
|
+
hold**, as in Cucumber: `--filter @smoke` takes a feature tagged
|
|
37
|
+
`@smoke`, and `not @wip` leaves out a Rule's scenarios, or an Examples
|
|
38
|
+
block's rows, tagged `@wip`. Reports, history and the IDEs' tag lists
|
|
39
|
+
carry them; a scenario lists every row's tags.
|
|
40
|
+
- **A scenario whose own name ends in "(example N)" keeps its results.**
|
|
41
|
+
Report and history rows now carry the scenario's name; an older row is
|
|
42
|
+
read as a scenario of that exact name first, and as an outline's row
|
|
43
|
+
only when no such scenario exists, in status, the dashboard and CI
|
|
44
|
+
results alike.
|
|
45
|
+
- **A run that writes an outline's recording runs every row of it**
|
|
46
|
+
(`--rerecord`, or an outline not recorded yet or whose recording is
|
|
47
|
+
stale), also rows an Examples line or a `--filter` on Examples tags left
|
|
48
|
+
out: the rows share one recording. A row asked for records first and
|
|
49
|
+
the others replay the new recording in the same run, so one that fits
|
|
50
|
+
only that row comes back red, not as a proposal.
|
|
51
|
+
- **The dashboard keeps each Examples row's latest result** from whichever
|
|
52
|
+
run ran it: a run that left a row out (a tag filter, a named run) no
|
|
53
|
+
longer clears that row's failure; a row removed from the table stops
|
|
54
|
+
counting.
|
|
55
|
+
- **A run of named scenarios is not a suite run.** Its results replace
|
|
56
|
+
those scenarios' own in the last report, beside the rest of the last
|
|
57
|
+
run, and the report takes the run's own times and hook results. Its history line is
|
|
58
|
+
marked partial: the dashboard counts it for those scenarios (their
|
|
59
|
+
latest result, never run, recent failures), while trends, the pass-rate
|
|
60
|
+
series, the last run's figures and the next run's comparison count runs
|
|
61
|
+
of the suite only. A last report that was imported from CI, for another
|
|
62
|
+
base URL or billed another way (subscription or API key) is replaced,
|
|
63
|
+
and the run says so. Sharded, such a run merges the same way.
|
|
64
|
+
- **Two scenarios of one name in one file stop the run**, as any two
|
|
65
|
+
scenarios sharing a recording do: one per Rule was allowed, and each
|
|
66
|
+
overwrote the other's recording. The check now reads the configured
|
|
67
|
+
`features` folder, not only `features/`, and always the files the run
|
|
68
|
+
names.
|
|
69
|
+
- **One pass rate everywhere.** The run report, the trends and `saffron
|
|
70
|
+
status --json` (a new `passRate` per run) leave out a heal whose proof
|
|
71
|
+
replay failed, as the dashboard already did: the same run no longer
|
|
72
|
+
reads 100% in the report and 0% on the dashboard.
|
|
73
|
+
- **History written on Windows counts on the dashboard.** A history line
|
|
74
|
+
naming `features\login.saffron` is the same feature as
|
|
75
|
+
`features/login.saffron`, so its scenarios keep their latest result and
|
|
76
|
+
recent failures instead of reading as never run.
|
|
77
|
+
- **A refused run deletes no recording.** `--rerecord` removes recordings
|
|
78
|
+
only when the scenarios are about to run: a run stopped first (a cache
|
|
79
|
+
collision, an `{env:...}` or `{data:...}` reference, an invalid
|
|
80
|
+
`--trace`, `--browser` or `--assertion-policy`, a failed setup hook, a
|
|
81
|
+
missing browser) keeps them.
|
|
82
|
+
- **A path that does not exist stops the run.** `saffron run
|
|
83
|
+
features/chekout.saffron` (a typo) exits 2 naming it, instead of
|
|
84
|
+
running the other paths alone.
|
|
85
|
+
- **Run lenses from the language server**, for an editor that asks for
|
|
86
|
+
them (`initializationOptions.runLenses`). `saffron lsp` puts Run, Replay
|
|
87
|
+
only and Headed above each scenario, in the file's own Gherkin
|
|
88
|
+
language, each carrying the scenario's `file:name` and the project to
|
|
89
|
+
run it in. A `.feature` file outside a Saffron project gets none, and
|
|
90
|
+
neither does a file outside the editor's workspace, so a lens never runs
|
|
91
|
+
anywhere the user did not open (such a file keeps its own project's
|
|
92
|
+
completion and diagnostics). A scenario named like lines (`42`,
|
|
93
|
+
`10:30`) is targeted by its line. The JetBrains plugin also refuses a
|
|
94
|
+
lens whose folder is outside the IDE project.
|
|
95
|
+
- `saffron status --json` gives each scenario's line from the parsed file:
|
|
96
|
+
right in another Gherkin language, and never a `Scenario:` line inside a
|
|
97
|
+
doc string.
|
|
98
|
+
- **Open Trace on a CI run works from the editors again.** `saffron trace
|
|
99
|
+
--run <id>` takes `--max-download <MB>`, as the JetBrains plugin passes
|
|
100
|
+
it, and refuses a larger download exactly as `report --run` does: exit
|
|
101
|
+
2, and with `--json` the same `kind: "too-large"` and `sizeBytes`.
|
|
102
|
+
- **Two editors can load one CI run at once.** Each request downloads
|
|
103
|
+
into its own temporary folder beside `.saffron/ci/<provider>-<id>/`, so
|
|
104
|
+
one never deletes another's download. When another request finished
|
|
105
|
+
first, Saffron uses its folder, checked as a cached one is, and removes
|
|
106
|
+
its own. A finished download removes only earlier attempts of the run,
|
|
107
|
+
never a later attempt or another request's download in progress; what
|
|
108
|
+
a killed download left (0.9.7's `.partial`, or a folder untouched for a
|
|
109
|
+
day) goes.
|
|
110
|
+
- **`--max-download` covers everything downloaded.** Saffron reads every
|
|
111
|
+
page of a GitHub run's artifacts, not just the first 100, and downloads
|
|
112
|
+
exactly the `saffron-bundle*` artifacts it sized, one by one by name, on
|
|
113
|
+
GitHub and Azure. An artifact whose name it does not accept is neither
|
|
114
|
+
counted nor downloaded.
|
|
115
|
+
- **Doc strings in `.saffron` files stay as written.** A line such as
|
|
116
|
+
`StepSet: sample` or `StepSet sample` inside a `"""` or ```` ``` ````
|
|
117
|
+
doc string is literal test data now, no longer read as a step set
|
|
118
|
+
definition or call.
|
|
119
|
+
|
|
120
|
+
VS Code extension 0.6.1
|
|
121
|
+
|
|
122
|
+
- **Health's pass rate of recent runs** is the runner's (saffron-ai 0.9.8
|
|
123
|
+
or later): a heal whose proof failed does not pass.
|
|
124
|
+
- **Open Report and Open Trace on a CI run keep `saffron.ciMaxDownloadMB`**
|
|
125
|
+
when the run's results are no longer on disk (trace with saffron-ai
|
|
126
|
+
0.9.8 or later).
|
|
127
|
+
- **Run, Replay only and Headed above each scenario.** A click runs that
|
|
128
|
+
scenario alone, by name (`saffron run <file>:<name>`), as a task in the
|
|
129
|
+
project that owns the file. The file is saved first, and the panel's
|
|
130
|
+
replay-only and headed toggles apply as they do to every run. Lenses
|
|
131
|
+
and the side panel find scenarios in the file's own Gherkin language
|
|
132
|
+
(`# language:`), nameless ones too. Needs saffron-ai 0.9.8 or later; with
|
|
133
|
+
an older one in the project a click says so.
|
|
134
|
+
- **A CI run's yellow results read "pending review", not "healed".**
|
|
135
|
+
Yellow also covers first recordings and proposals not yet verified.
|
|
136
|
+
- **Each yellow or red CI scenario says whether the run verified it**:
|
|
137
|
+
"unverified" or "verified", and nothing when the run does not say.
|
|
138
|
+
|
|
139
|
+
JetBrains plugin 0.6.1
|
|
140
|
+
|
|
141
|
+
- **Health's pass rate of recent runs** is the runner's (saffron-ai 0.9.8
|
|
142
|
+
or later): a heal whose proof failed does not pass.
|
|
143
|
+
- **Run, Replay only and Headed above each scenario**, with LSP4IJ
|
|
144
|
+
installed. A click saves the files and runs that scenario alone, by
|
|
145
|
+
name, through a temporary run configuration made for it, so the run
|
|
146
|
+
widget runs it again. A saved configuration of the same name is left as
|
|
147
|
+
it is, and nothing from the Saffron template carries in, and a scenario
|
|
148
|
+
of the same name in another file gets its own. The lenses work while the
|
|
149
|
+
IDE indexes, and say when the project's saffron-ai is too old.
|
|
150
|
+
- **Right-click Run inside a scenario** runs that scenario alone, with or
|
|
151
|
+
without LSP4IJ, never reusing a lens's Replay only or Headed
|
|
152
|
+
configuration; outside one, or in a file in another Gherkin language,
|
|
153
|
+
it runs the whole file, as before. Both need saffron-ai 0.9.8 or later.
|
|
154
|
+
- **Open Trace works with every runner and keeps the download limit.**
|
|
155
|
+
With saffron-ai 0.9.8 it passes `--max-download` to `saffron trace`,
|
|
156
|
+
which downloads a run no longer on disk; with 0.9.7, which rejects the
|
|
157
|
+
option, it leaves it out.
|
|
158
|
+
- **Open Trace finds a Scenario Outline's example row** ("Login (example
|
|
159
|
+
1)") by the name the run shows.
|
|
160
|
+
- **CI yellow results read "pending review", not "healed"**, and each
|
|
161
|
+
scenario says "unverified" or "verified" when the run reports it.
|
|
162
|
+
|
|
8
163
|
## 0.9.7: CI results without importing
|
|
9
164
|
|
|
10
165
|
Released 2026-10-05.
|
package/README.md
CHANGED
|
@@ -73,7 +73,7 @@ node dist/cli/index.js -p examples accept --all
|
|
|
73
73
|
|
|
74
74
|
| Command | What it does |
|
|
75
75
|
|---|---|
|
|
76
|
-
| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
|
|
76
|
+
| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
|
|
77
77
|
| `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
|
|
78
78
|
| `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
|
|
79
79
|
| `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
|
|
@@ -155,8 +155,7 @@ milestones, statuses and decision gates, and everything shipped so far in
|
|
|
155
155
|
its "Where we are" table.
|
|
156
156
|
Headlines ahead: API testing (reusable REST operations, fixtures and schema
|
|
157
157
|
validation), scenarios with two users in two browsers, a configurable
|
|
158
|
-
features folder,
|
|
159
|
-
IDEs), parallel recording and a browser matrix in one run, failure video,
|
|
158
|
+
features folder, parallel recording and a browser matrix in one run, failure video,
|
|
160
159
|
agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
|
|
161
160
|
(gated on Level-1 divergence data), and the long-term open-core SaaS layer.
|
|
162
161
|
|