saffron-ai 0.9.7 → 0.9.9

package/CHANGELOG.md

@@ -5,6 +5,223 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
 release heading names its main feature: `## <version>: <title>`.
 
+## 0.9.9: A browser matrix in one run
+
+Released 2026-10-06.
+
+Runner
+
+- **A browser matrix in one run.** `saffron run --browser
+  chromium,firefox,webkit` (or a list in `browser`) runs the suite in each
+  browser, one after another, into one report: a row per scenario per
+  browser, in the terminal, the report, the history and the dashboard.
+  Chromium-family browsers go first, so Firefox and WebKit replay what
+  they recorded. Every browser starts before anything runs. Screenshots
+  and traces are kept per browser, and `saffron trace` names them
+  `feature:scenario@browser`. The dashboard stands a scenario at its
+  worst browser and never calls a difference between browsers flaky.
+  Refused with `--shard`. Every result now names its browser, so running
+  one browser again replaces that browser's rows and keeps the others'.
+  `saffron status --json` (and the MCP `project_status` tool) names the
+  browser of each row that needs attention.
+- **Device and viewport profiles.** `--device "iPhone 15"` (config
+  `device`) runs every scenario under a Playwright device profile,
+  recordings and heals included; `--viewport 1280x720` (config `viewport`)
+  sets the size alone. An unknown device names the closest ones, and a
+  mobile profile in Firefox stops the run up front.
+- **Parallel recording and healing.** `--agent-workers N` (config
+  `agentWorkers`, default 1) lets up to N agent sessions run at once,
+  each in a browser of its own. An outline's rows stay together on one
+  worker. Each session is paid for: raise it deliberately.
+- **Shards know their browser.** A shard records the browser it ran in;
+  `saffron report --merge` refuses shards of different browsers, a
+  re-run attempt replaces only its own browser's shard, and each
+  browser's merge keeps its own history line.
+- **A bad storage state stops the run first.** A `storageState` file that
+  is missing, unreadable or not JSON now stops the run at startup, exit 2,
+  before `--rerecord` deletes any recording.
+- **No second replay without an agent.** With `--workers` and
+  `--no-agent`, or in Firefox and WebKit, a scenario that went red in the
+  parallel phase is no longer run again afterwards: nothing could change
+  its result.
+- **Per-worker timing.** The report has a timing panel (each worker's
+  runs, busy time and share, and the five slowest scenarios), and the
+  terminal prints each worker's share and the three slowest scenarios.
+  Merged shards list each machine's workers.
+- **Saffron's own data in the replay.** Each step shows the tabs it
+  opened or switched to and the response each network check matched,
+  and the Saffron pane shows what the agent cost for the scenario. The
+  report's steps carry `tabs` and `responses` for tools.
+- **Browsers in the editors' contracts.** `saffron status --json` and
+  `saffron report --run <id> --json` give each row its `browser`, and the
+  run's `browsers` when its rows span several.
+
+VS Code extension 0.6.2
+
+- After a browser-matrix run, Last Run and a CI run's results label each
+  row with its browser, and Open Replay and Open Trace open that
+  browser's execution alone (`feature:scenario@browser`).
+
+JetBrains plugin 0.6.2
+
+- The same in the Last Run and CI Runs tabs: rows say their browser, and
+  the replay opens that browser's execution alone.
+
+## 0.9.8: Run one scenario, from the editor too
+
+Released 2026-10-06.
+
+Behaviour changes
+
+- **Tags now apply as in Cucumber**: a Feature's and a Rule's tags to
+  every scenario in it, an Examples block's to its rows. A CI filter can
+  select differently than before: `not @ignore` with `@ignore` on a
+  Feature now skips that feature, and `--filter @smoke` with a
+  Feature-level `@smoke` now takes all of its scenarios. Check your
+  `--filter` expressions when you upgrade. The dashboard's results per tag
+  and the IDEs' Tags tabs count the inherited tags too.
+
+Runner
+
+- **Run one scenario by its line or its name.** `saffron run
+  features/login.saffron:12` runs the scenario that line is in,
+  Cucumber-style: its Scenario line, a tag above it or any line inside it.
+  An Examples row runs that row alone, any other line of an outline runs
+  every row, and `:12:30` names two. `features/login.saffron:Successful
+  login` names it instead, which stays true while the file is edited,
+  and `:Roles (example 2)` one Examples row, as `saffron trace` lists it. A
+  line no scenario owns (the feature's header, a Background, a StepSet, a
+  Rule's header, past the end), a name no scenario has, or an outline
+  whose Examples have no rows yet stops the run with exit 2 and says
+  what is there. Works with `--filter`, `--shard` and `--rerecord`.
+- **Tags on a Feature, a Rule or an Examples block apply to what they
+  hold**, as in Cucumber: `--filter @smoke` takes a feature tagged
+  `@smoke`, and `not @wip` leaves out a Rule's scenarios, or an Examples
+  block's rows, tagged `@wip`. Reports, history and the IDEs' tag lists
+  carry them; a scenario lists every row's tags.
+- **A scenario whose own name ends in "(example N)" keeps its results.**
+  Report and history rows now carry the scenario's name; an older row is
+  read as a scenario of that exact name first, and as an outline's row
+  only when no such scenario exists, in status, the dashboard and CI
+  results alike.
+- **A run that writes an outline's recording runs every row of it**
+  (`--rerecord`, or an outline not recorded yet or whose recording is
+  stale), also rows an Examples line or a `--filter` on Examples tags left
+  out: the rows share one recording. A row asked for records first and
+  the others replay the new recording in the same run, so one that fits
+  only that row comes back red, not as a proposal.
+- **The dashboard keeps each Examples row's latest result** from whichever
+  run ran it: a run that left a row out (a tag filter, a named run) no
+  longer clears that row's failure; a row removed from the table stops
+  counting.
+- **A run of named scenarios is not a suite run.** Its results replace
+  those scenarios' own in the last report, beside the rest of the last
+  run, and the report takes the run's own times and hook results. Its history line is
+  marked partial: the dashboard counts it for those scenarios (their
+  latest result, never run, recent failures), while trends, the pass-rate
+  series, the last run's figures and the next run's comparison count runs
+  of the suite only. A last report that was imported from CI, for another
+  base URL or billed another way (subscription or API key) is replaced,
+  and the run says so. Sharded, such a run merges the same way.
+- **Two scenarios of one name in one file stop the run**, as any two
+  scenarios sharing a recording do: one per Rule was allowed, and each
+  overwrote the other's recording. The check now reads the configured
+  `features` folder, not only `features/`, and always the files the run
+  names.
+- **One pass rate everywhere.** The run report, the trends and `saffron
+  status --json` (a new `passRate` per run) leave out a heal whose proof
+  replay failed, as the dashboard already did: the same run no longer
+  reads 100% in the report and 0% on the dashboard.
+- **History written on Windows counts on the dashboard.** A history line
+  naming `features\login.saffron` is the same feature as
+  `features/login.saffron`, so its scenarios keep their latest result and
+  recent failures instead of reading as never run.
+- **A refused run deletes no recording.** `--rerecord` removes recordings
+  only when the scenarios are about to run: a run stopped first (a cache
+  collision, an `{env:...}` or `{data:...}` reference, an invalid
+  `--trace`, `--browser` or `--assertion-policy`, a failed setup hook, a
+  missing browser) keeps them.
+- **A path that does not exist stops the run.** `saffron run
+  features/chekout.saffron` (a typo) exits 2 naming it, instead of
+  running the other paths alone.
+- **Run lenses from the language server**, for an editor that asks for
+  them (`initializationOptions.runLenses`). `saffron lsp` puts Run, Replay
+  only and Headed above each scenario, in the file's own Gherkin
+  language, each carrying the scenario's `file:name` and the project to
+  run it in. A `.feature` file outside a Saffron project gets none, and
+  neither does a file outside the editor's workspace, so a lens never runs
+  anywhere the user did not open (such a file keeps its own project's
+  completion and diagnostics). A scenario named like lines (`42`,
+  `10:30`) is targeted by its line. The JetBrains plugin also refuses a
+  lens whose folder is outside the IDE project.
+- `saffron status --json` gives each scenario's line from the parsed file:
+  right in another Gherkin language, and never a `Scenario:` line inside a
+  doc string.
+- **Open Trace on a CI run works from the editors again.** `saffron trace
+  --run <id>` takes `--max-download <MB>`, as the JetBrains plugin passes
+  it, and refuses a larger download exactly as `report --run` does: exit
+  2, and with `--json` the same `kind: "too-large"` and `sizeBytes`.
+- **Two editors can load one CI run at once.** Each request downloads
+  into its own temporary folder beside `.saffron/ci/<provider>-<id>/`, so
+  one never deletes another's download. When another request finished
+  first, Saffron uses its folder, checked as a cached one is, and removes
+  its own. A finished download removes only earlier attempts of the run,
+  never a later attempt or another request's download in progress; what
+  a killed download left (0.9.7's `.partial`, or a folder untouched for a
+  day) goes.
+- **`--max-download` covers everything downloaded.** Saffron reads every
+  page of a GitHub run's artifacts, not just the first 100, and downloads
+  exactly the `saffron-bundle*` artifacts it sized, one by one by name, on
+  GitHub and Azure. An artifact whose name it does not accept is neither
+  counted nor downloaded.
+- **Doc strings in `.saffron` files stay as written.** A line such as
+  `StepSet: sample` or `StepSet sample` inside a `"""` or ```` ``` ````
+  doc string is literal test data now, no longer read as a step set
+  definition or call.
+
+VS Code extension 0.6.1
+
+- **Health's pass rate of recent runs** is the runner's (saffron-ai 0.9.8
+  or later): a heal whose proof failed does not pass.
+- **Open Report and Open Trace on a CI run keep `saffron.ciMaxDownloadMB`**
+  when the run's results are no longer on disk (trace with saffron-ai
+  0.9.8 or later).
+- **Run, Replay only and Headed above each scenario.** A click runs that
+  scenario alone, by name (`saffron run <file>:<name>`), as a task in the
+  project that owns the file. The file is saved first, and the panel's
+  replay-only and headed toggles apply as they do to every run. Lenses
+  and the side panel find scenarios in the file's own Gherkin language
+  (`# language:`), nameless ones too. Needs saffron-ai 0.9.8 or later; with
+  an older one in the project a click says so.
+- **A CI run's yellow results read "pending review", not "healed".**
+  Yellow also covers first recordings and proposals not yet verified.
+- **Each yellow or red CI scenario says whether the run verified it**:
+  "unverified" or "verified", and nothing when the run does not say.
+
+JetBrains plugin 0.6.1
+
+- **Health's pass rate of recent runs** is the runner's (saffron-ai 0.9.8
+  or later): a heal whose proof failed does not pass.
+- **Run, Replay only and Headed above each scenario**, with LSP4IJ
+  installed. A click saves the files and runs that scenario alone, by
+  name, through a temporary run configuration made for it, so the run
+  widget runs it again. A saved configuration of the same name is left as
+  it is, and nothing from the Saffron template carries in, and a scenario
+  of the same name in another file gets its own. The lenses work while the
+  IDE indexes, and say when the project's saffron-ai is too old.
+- **Right-click Run inside a scenario** runs that scenario alone, with or
+  without LSP4IJ, never reusing a lens's Replay only or Headed
+  configuration; outside one, or in a file in another Gherkin language,
+  it runs the whole file, as before. Both need saffron-ai 0.9.8 or later.
+- **Open Trace works with every runner and keeps the download limit.**
+  With saffron-ai 0.9.8 it passes `--max-download` to `saffron trace`,
+  which downloads a run no longer on disk; with 0.9.7, which rejects the
+  option, it leaves it out.
+- **Open Trace finds a Scenario Outline's example row** ("Login (example
+  1)") by the name the run shows.
+- **CI yellow results read "pending review", not "healed"**, and each
+  scenario says "unverified" or "verified" when the run reports it.
+
 ## 0.9.7: CI results without importing
 
 Released 2026-10-05.

package/README.md

@@ -73,7 +73,7 @@ node dist/cli/index.js -p examples accept --all
 
 | Command | What it does |
 |---|---|
-| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
+| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b[,b...]] [--device name] [--viewport WxH] [--no-hooks] [--workers n] [--agent-workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
 | `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
 | `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
@@ -155,8 +155,7 @@ milestones, statuses and decision gates, and everything shipped so far in
 its "Where we are" table.
 Headlines ahead: API testing (reusable REST operations, fixtures and schema
 validation), scenarios with two users in two browsers, a configurable
-features folder, running one scenario (`file:line`, with run icons in both
-IDEs), parallel recording and a browser matrix in one run, failure video,
+features folder, failure video,
 agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
 (gated on Level-1 divergence data), and the long-term open-core SaaS layer.