saffron-ai 0.9.6 → 0.9.8

package/CHANGELOG.md

@@ -5,6 +5,236 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
 release heading names its main feature: `## <version>: <title>`.
 
+## 0.9.8: Run one scenario, from the editor too
+
+Released 2026-10-06.
+
+Behaviour changes
+
+- **Tags now apply as in Cucumber**: a Feature's and a Rule's tags to
+  every scenario in it, an Examples block's to its rows. A CI filter can
+  select differently than before: `not @ignore` with `@ignore` on a
+  Feature now skips that feature, and `--filter @smoke` with a
+  Feature-level `@smoke` now takes all of its scenarios. Check your
+  `--filter` expressions when you upgrade. The dashboard's results per tag
+  and the IDEs' Tags tabs count the inherited tags too.
+
+Runner
+
+- **Run one scenario by its line or its name.** `saffron run
+  features/login.saffron:12` runs the scenario that line is in,
+  Cucumber-style: its Scenario line, a tag above it or any line inside it.
+  An Examples row runs that row alone, any other line of an outline runs
+  every row, and `:12:30` names two. `features/login.saffron:Successful
+  login` names it instead, which stays true while the file is edited,
+  and `:Roles (example 2)` one Examples row, as `saffron trace` lists it. A
+  line no scenario owns (the feature's header, a Background, a StepSet, a
+  Rule's header, past the end), a name no scenario has, or an outline
+  whose Examples have no rows yet stops the run with exit 2 and says
+  what is there. Works with `--filter`, `--shard` and `--rerecord`.
+- **Tags on a Feature, a Rule or an Examples block apply to what they
+  hold**, as in Cucumber: `--filter @smoke` takes a feature tagged
+  `@smoke`, and `not @wip` leaves out a Rule's scenarios, or an Examples
+  block's rows, tagged `@wip`. Reports, history and the IDEs' tag lists
+  carry them; a scenario lists every row's tags.
+- **A scenario whose own name ends in "(example N)" keeps its results.**
+  Report and history rows now carry the scenario's name; an older row is
+  read as a scenario of that exact name first, and as an outline's row
+  only when no such scenario exists, in status, the dashboard and CI
+  results alike.
+- **A run that writes an outline's recording runs every row of it**
+  (`--rerecord`, or an outline not recorded yet or whose recording is
+  stale), also rows an Examples line or a `--filter` on Examples tags left
+  out: the rows share one recording. A row asked for records first and
+  the others replay the new recording in the same run, so one that fits
+  only that row comes back red, not as a proposal.
+- **The dashboard keeps each Examples row's latest result** from whichever
+  run ran it: a run that left a row out (a tag filter, a named run) no
+  longer clears that row's failure; a row removed from the table stops
+  counting.
+- **A run of named scenarios is not a suite run.** Its results replace
+  those scenarios' own in the last report, beside the rest of the last
+  run, and the report takes the run's own times and hook results. Its history line is
+  marked partial: the dashboard counts it for those scenarios (their
+  latest result, never run, recent failures), while trends, the pass-rate
+  series, the last run's figures and the next run's comparison count runs
+  of the suite only. A last report that was imported from CI, for another
+  base URL or billed another way (subscription or API key) is replaced,
+  and the run says so. Sharded, such a run merges the same way.
+- **Two scenarios of one name in one file stop the run**, as any two
+  scenarios sharing a recording do: one per Rule was allowed, and each
+  overwrote the other's recording. The check now reads the configured
+  `features` folder, not only `features/`, and always the files the run
+  names.
+- **One pass rate everywhere.** The run report, the trends and `saffron
+  status --json` (a new `passRate` per run) leave out a heal whose proof
+  replay failed, as the dashboard already did: the same run no longer
+  reads 100% in the report and 0% on the dashboard.
+- **History written on Windows counts on the dashboard.** A history line
+  naming `features\login.saffron` is the same feature as
+  `features/login.saffron`, so its scenarios keep their latest result and
+  recent failures instead of reading as never run.
+- **A refused run deletes no recording.** `--rerecord` removes recordings
+  only when the scenarios are about to run: a run stopped first (a cache
+  collision, an `{env:...}` or `{data:...}` reference, an invalid
+  `--trace`, `--browser` or `--assertion-policy`, a failed setup hook, a
+  missing browser) keeps them.
+- **A path that does not exist stops the run.** `saffron run
+  features/chekout.saffron` (a typo) exits 2 naming it, instead of
+  running the other paths alone.
+- **Run lenses from the language server**, for an editor that asks for
+  them (`initializationOptions.runLenses`). `saffron lsp` puts Run, Replay
+  only and Headed above each scenario, in the file's own Gherkin
+  language, each carrying the scenario's `file:name` and the project to
+  run it in. A `.feature` file outside a Saffron project gets none, and
+  neither does a file outside the editor's workspace, so a lens never runs
+  anywhere the user did not open (such a file keeps its own project's
+  completion and diagnostics). A scenario named like lines (`42`,
+  `10:30`) is targeted by its line. The JetBrains plugin also refuses a
+  lens whose folder is outside the IDE project.
+- `saffron status --json` gives each scenario's line from the parsed file:
+  right in another Gherkin language, and never a `Scenario:` line inside a
+  doc string.
+- **Open Trace on a CI run works from the editors again.** `saffron trace
+  --run <id>` takes `--max-download <MB>`, as the JetBrains plugin passes
+  it, and refuses a larger download exactly as `report --run` does: exit
+  2, and with `--json` the same `kind: "too-large"` and `sizeBytes`.
+- **Two editors can load one CI run at once.** Each request downloads
+  into its own temporary folder beside `.saffron/ci/<provider>-<id>/`, so
+  one never deletes another's download. When another request finished
+  first, Saffron uses its folder, checked as a cached one is, and removes
+  its own. A finished download removes only earlier attempts of the run,
+  never a later attempt or another request's download in progress; what
+  a killed download left (0.9.7's `.partial`, or a folder untouched for a
+  day) goes.
+- **`--max-download` covers everything downloaded.** Saffron reads every
+  page of a GitHub run's artifacts, not just the first 100, and downloads
+  exactly the `saffron-bundle*` artifacts it sized, one by one by name, on
+  GitHub and Azure. An artifact whose name it does not accept is neither
+  counted nor downloaded.
+- **Doc strings in `.saffron` files stay as written.** A line such as
+  `StepSet: sample` or `StepSet sample` inside a `"""` or ```` ``` ````
+  doc string is literal test data now, no longer read as a step set
+  definition or call.
+
+VS Code extension 0.6.1
+
+- **Health's pass rate of recent runs** is the runner's (saffron-ai 0.9.8
+  or later): a heal whose proof failed does not pass.
+- **Open Report and Open Trace on a CI run keep `saffron.ciMaxDownloadMB`**
+  when the run's results are no longer on disk (trace with saffron-ai
+  0.9.8 or later).
+- **Run, Replay only and Headed above each scenario.** A click runs that
+  scenario alone, by name (`saffron run <file>:<name>`), as a task in the
+  project that owns the file. The file is saved first, and the panel's
+  replay-only and headed toggles apply as they do to every run. Lenses
+  and the side panel find scenarios in the file's own Gherkin language
+  (`# language:`), nameless ones too. Needs saffron-ai 0.9.8 or later; with
+  an older one in the project a click says so.
+- **A CI run's yellow results read "pending review", not "healed".**
+  Yellow also covers first recordings and proposals not yet verified.
+- **Each yellow or red CI scenario says whether the run verified it**:
+  "unverified" or "verified", and nothing when the run does not say.
+
+JetBrains plugin 0.6.1
+
+- **Health's pass rate of recent runs** is the runner's (saffron-ai 0.9.8
+  or later): a heal whose proof failed does not pass.
+- **Run, Replay only and Headed above each scenario**, with LSP4IJ
+  installed. A click saves the files and runs that scenario alone, by
+  name, through a temporary run configuration made for it, so the run
+  widget runs it again. A saved configuration of the same name is left as
+  it is, and nothing from the Saffron template carries in, and a scenario
+  of the same name in another file gets its own. The lenses work while the
+  IDE indexes, and say when the project's saffron-ai is too old.
+- **Right-click Run inside a scenario** runs that scenario alone, with or
+  without LSP4IJ, never reusing a lens's Replay only or Headed
+  configuration; outside one, or in a file in another Gherkin language,
+  it runs the whole file, as before. Both need saffron-ai 0.9.8 or later.
+- **Open Trace works with every runner and keeps the download limit.**
+  With saffron-ai 0.9.8 it passes `--max-download` to `saffron trace`,
+  which downloads a run no longer on disk; with 0.9.7, which rejects the
+  option, it leaves it out.
+- **Open Trace finds a Scenario Outline's example row** ("Login (example
+  1)") by the name the run shows.
+- **CI yellow results read "pending review", not "healed"**, and each
+  scenario says "unverified" or "verified" when the run reports it.
+
+## 0.9.7: CI results without importing
+
+Released 2026-10-05.
+
+Runner
+
+- **See what a CI run found before you import it.** `saffron report --run
+  <id>` downloads the run's Saffron bundles once into
+  `.saffron/ci/<provider>-<id>/attempt-<n>/`, checks them as `saffron
+  import` does (repository, digests, sizes, format, the run, attempts and
+  shards), judges each proposal against this checkout without filing it,
+  and opens the run's report page. Your `.saffron/reports/`,
+  `.saffron/proposals/`, `.saffron/artifacts/` and import ledger are left
+  as they were. `--json` gives the run, its totals, each scenario with its
+  failing step, first error line, screenshot and trace, each proposal as
+  importable or not with the reason, and warnings: what the VS Code and
+  JetBrains CI runs views show under a run.
+- **A run without results says why.** `running` (not finished),
+  `expired` (the service deleted the artifact) or `none` (nothing
+  uploaded; the guide's workflow exports only on failure, and `if:
+  always()` on the export and upload steps keeps every run's results), as
+  `reasonKind` and `reason`, exit 0.
+- **Large runs ask first.** `--max-download <MB>` refuses a run whose
+  bundles add up to more, with `{"error", "kind": "too-large",
+  "sizeBytes"}` and exit 2, so an editor can ask and run again.
+- **Replay a CI run, view only.** `saffron trace --run <id> [scenario]`
+  replays the run's traces from `.saffron/ci/`, taking the same scenario
+  forms as `saffron trace` (`<feature>:<scenario>` included). The page has
+  no Accept or Reject and its server no review routes: nothing was filed.
+- **View, then import, downloads once.** `saffron import --run <id>`
+  reuses the viewed download of the run's latest attempt.
+- **A cache that looks after itself.** `.saffron/ci/` keeps the newest 10
+  runs and at most 1 GB, removing the oldest first; a download goes
+  through a `.partial` folder and is renamed whole, so a cancelled one
+  leaves nothing behind. The folder writes its own `.gitignore` (`*`), so
+  it never shows in `git status`, even in a project set up before it
+  existed, and `saffron init` adds `.saffron/ci/` to the entries it
+  manages.
+
+VS Code extension 0.6.0
+
+- **Expand a CI run to see what it found, without importing it.** The
+  run lists its totals, each failed and healed scenario with its failing
+  step and first error line (the tooltip has what the agent changed), and
+  "N proposals: M importable" with the reason each one would not import.
+  Your reports, proposals and evidence are left alone. Needs saffron-ai
+  0.9.7 or later; an older runner says so.
+- **Open Report and Open Trace on a viewed run.** Open Report shows the
+  run's report in your browser; Open Trace replays a traced scenario,
+  view only. Import stays the only action that files proposals.
+- **Large results ask first.** Results above the new
+  `saffron.ciMaxDownloadMB` setting (default 100, 0 for no limit) ask
+  "This run's results are N MB. Download?". Loading can be cancelled,
+  which stops the whole download, and never runs while the CI runs view
+  is hidden.
+- **A run with no results says why:** still running, its artifacts
+  expired, or it uploaded none.
+
+JetBrains plugin 0.6.0
+
+- **CI Runs shows what a run found without importing it.** Select a run
+  to see its totals, each failed and healed scenario with its failing
+  step and first error line (the narrative in the tooltip), and
+  "N proposals: M importable" with the reason for each one that is not.
+  It loads in a background task you can cancel, only while the tab is on
+  screen. Needs saffron-ai 0.9.7 or later; an older runner says so.
+- **Open Report and Open Trace on a viewed run.** Open Trace (or a
+  double-click) replays a failed or healed scenario's trace, view only.
+  Import Run stays the only action that brings proposals into the
+  checkout.
+- **Large results ask first**, above the limit in Settings, Tools, Saffron
+  (default 100 MB, 0 for no limit), the same as the VS Code setting.
+- **A run with no results says why:** still running, expired, or none
+  uploaded.
+
 ## 0.9.6: Dashboard figures you can trust
 
 Released 2026-10-05.

package/README.md

@@ -73,7 +73,7 @@ node dist/cli/index.js -p examples accept --all
 
 | Command | What it does |
 |---|---|
-| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
+| `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). A path with a line or a name, `features/login.saffron:12` or `"features/login.saffron:Successful login"`, runs that one scenario. Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
 | `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
 | `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
@@ -155,8 +155,7 @@ milestones, statuses and decision gates, and everything shipped so far in
 its "Where we are" table.
 Headlines ahead: API testing (reusable REST operations, fixtures and schema
 validation), scenarios with two users in two browsers, a configurable
-features folder, running one scenario (`file:line`, with run icons in both
-IDEs), parallel recording and a browser matrix in one run, failure video,
+features folder, parallel recording and a browser matrix in one run, failure video,
 agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
 (gated on Level-1 divergence data), and the long-term open-core SaaS layer.