saffron-ai 0.8.5 → 0.9.0

package/CHANGELOG.md

@@ -4,6 +4,201 @@ All notable changes to Saffron, in one place: the `saffron-ai` runner (npm),
 the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates.
 
+## 0.9.0
+
+Released 2026-09-27.
+
+Runner
+
+- **Execution replay: `saffron trace`.** Not a video. With `"trace":
+  "retain-on-failure"` (or `"on"`) in the config, or `--trace <mode>` for
+  one run, a scenario keeps an execution trace, and `saffron trace` opens
+  it as a replay in your browser: your Given, When and Then on the left,
+  each step opening onto the actions Saffron replayed and the Playwright
+  calls behind them; the page at the selected action in the middle,
+  rendered as a live DOM you can inspect, before, during or after the
+  action, with a filmstrip of the whole run; on the right the failure, the
+  retries, the call log, the network during the selection, and what the
+  agent changed. The failed step is selected when you arrive. Traces live
+  next to the screenshots under `.saffron/artifacts/`, are kept for the
+  latest run and pruned with the rest. The report names the command for
+  every traced scenario, as `feature:scenario`. The page opens on `127.0.0.1` or `localhost`, a
+  forwarded port included, and keeps what it loaded when a later run
+  replaces the files. A tab left open from before `saffron trace` was
+  started again says to reload the page instead of loading the new run's
+  files. The trace is Playwright's own format, rendered by
+  the snapshot engine of the Playwright Saffron runs with; the replay
+  around it is Saffron's. The Saucedemo example config ships with
+  `retain-on-failure`.
+- **Secrets in a trace.** `{env:VAR}` values, including one a data file
+  holds, are masked back to their tokens before the trace is kept: in the
+  event log, the DOM snapshots and stylesheets, and every request and
+  response body, URL, header and WebSocket message, in the spellings the
+  value takes there (JSON escapes, percent-encoding as browsers and
+  servers write it, HTML entities, CSS escapes). The unmasked trace never
+  reaches `.saffron/artifacts/`, even when a run is killed while saving
+  it. The scheme, host and port of a URL are kept, so the replay can
+  still load the page's stylesheets and images; a secret anywhere else in
+  it, credentials included, is written `%7Benv:NAME%7D`. A value
+  shorter than three characters, one inside base64 or another binary
+  encoding, a value in a URL's host, and the filmstrip's pixels are not
+  masked, so look through a trace before attaching it to a public issue.
+- **Review a heal from the replay.** The selected step shows what the
+  proposal changes for it, the elements Playwright actually matched are
+  outlined in the page, and the Saffron panel has **Accept recording** and
+  **Reject proposal**. Both act only on the exact proposal associated with
+  this run: if it was replaced, accepted or rejected since, the replay
+  refuses and says why. One that went stale against the scenario or the
+  committed recording, or can no longer be read, cannot be accepted but
+  can still be rejected. An unverified recording, or one this run showed
+  failing, needs an explicit acknowledgement first. Suggested feature
+  rewrites are listed with the command that applies them (`saffron accept
+  --with-feature-edit`); they and propagation to other recordings stay
+  with `saffron accept`.
+- **Accept or reject the proposal you listed.** `saffron status --json`
+  gives each pending proposal a `revision`, and `saffron accept
+  <file>#<revision>` or `saffron reject <file>#<revision>` decides that
+  proposal only if it is still the one listed. One a later run replaced
+  is refused ("changed since it was listed. Refresh and review it
+  again."), exit 1, and left in place. A decision and a run filing a
+  proposal never interleave, so what is checked is what is promoted, with
+  or without a revision. Accept and reject act only on files under
+  `.saffron/proposals`: `saffron reject features/login.saffron` used to
+  delete the feature file.
+- **Proof replays and the agent in the trace.** A recorded or healed
+  scenario also keeps a trace of each proof replay (`proof-1.zip`,
+  `proof-2.zip`) with its own step outcomes, and **Trace** at the top of
+  the page switches between them and the run itself. The agent's own
+  browser calls are recorded too, under the step it was working on and in
+  words ("click Login button", "read the page"); what it looked at before
+  naming a step is filed under the step it named next. A healed step is
+  marked ✦, and a healed run opens on it.
+  If that cannot start with the installed Playwright, Saffron says why,
+  once, and the agent works through the standard browser connection
+  without being traced.
+- **Masking fixes outside the trace.** A secret whose value appears in
+  another variable's name or in a `{data:...}` key no longer corrupts the
+  recording (`ROLE=ADMIN` beside `{env:ADMIN_PASSWORD}` used to be
+  recorded as `{env:{env:ROLE}_PASSWORD}`, which never replayed).
+  Three-character values (a CVV, a short PIN) are now masked in
+  recordings, reports and history too, as in the trace; in a recording a
+  value shorter than six characters is masked where it stands alone or
+  where the step joins it to other text (`"qa{env:ID}"`), so `ENV=dev`
+  never turns `/devices` into `/{env:ENV}ices`. Reports mask
+  a value's URL-encoded and HTML-escaped forms. A recording masks a
+  secret in every field replay reads (labels, placeholders, test ids,
+  fallback selectors, frame names, drag targets, URL patterns, the text
+  for a prompt dialog), not only typed values and URLs, and replay
+  resolves the token in each of them: a selector recorded as
+  `td.{env:NAME}` still finds its element. A
+  `{env:VAR}` written as a value in a data file is masked in recordings
+  and reports like one written in the step.
+- **The HTML report embeds only the run's own screenshots.** A merged
+  report (`saffron report --merge`) could be made to embed any file the
+  merging machine could read, such as `.env`; it now embeds pictures from
+  `.saffron/artifacts/` only, and escapes the setup and teardown lines.
+- **Evidence from an interrupted run is not mixed in.** Screenshots and
+  traces are kept at fixed paths that the next run replaces. A run
+  stopped before it wrote its report used to leave the previous report
+  pointing at the new files; the IDE panels now leave out such a
+  screenshot and `saffron trace` refuses such a trace.
+- **Playwright 1.63 and trace format 9.** The runner now depends on
+  Playwright 1.63, and the replay reads both trace formats: 8 (up to
+  1.62) and 9 (1.63 and later), which files snapshots under the call by
+  phase and screencast frames under their own folder. A project that
+  resolved 1.63 saw broken filmstrip tiles and "No snapshot for this
+  selection".
+- **The first snapshot shown no longer arrives unstyled.** The snapshot
+  engine forgets a trace while its first frame is still loading, so that
+  frame's stylesheets and fonts came back missing and the page looked
+  bare until the next step was selected. The replay now warms the engine
+  with a hidden frame first.
+- **A step with no snapshot of its own shows the last page there was.** A
+  step the agent only inspected (the accessibility tree, the title, the
+  console) leaves no DOM snapshot, since Playwright takes them around
+  interactions. Instead of an empty pane, the replay shows the page after
+  the last interaction before that step and says so above it. A step
+  that never ran, after a failure, shows no page: "This step was never
+  reached." An action
+  that took no snapshot (reading the title, say) shows the page as the
+  last snapshot before it left it, not as its step began. With several
+  tabs open it only borrows from the same tab; Playwright 1.63 does not
+  record which tab such an action ran in, so when that is unknown the
+  replay says so, and the filmstrip marks no frame, rather than show
+  another tab. A tab opened later in the run changes nothing before it, and one
+  closed changes nothing after.
+- **Every traced scenario in one replay.** `saffron trace` with no name
+  (or a name several scenarios share) opens them all, with a Scenario
+  picker in the header, starting on a red one, else a yellow one. A run
+  with `"trace": "on"` where everything passed opens too. `--json` with no
+  name prints the scenario the replay opens on. Other scenarios' traces
+  are read when first picked and checked against their digest; one that
+  cannot be read says why.
+- **A step lists what happened in the order it happened.** Saffron's page
+  check, the recorded actions, the agent's and the calls outside any step
+  are interleaved by time, and the arrow keys follow that order. A call
+  outside every step is filed under the step running when it started. The
+  keys keep working after a click on a button or tab, leave Cmd, Ctrl and
+  Alt shortcuts to the browser, and the selected row scrolls into view.
+- **✦ marks only a step the agent healed.** In a first recording, a step
+  the agent worked through reads "recorded by the agent". In a proof
+  replay, or a run that replayed a pending proposal without the agent, a
+  step the proposal changes reads "ran the proposed recording". Neither
+  is called healed, and Details names the kind of run: Heal review, First
+  recording, New recording or Proposed recording.
+- **A pending proposal that failed is shown apart.** When a run replays a
+  pending proposal, it fails, and the run goes on to the committed
+  recording or records again, the proposal's attempt is one section,
+  "Earlier attempt: the pending proposal", instead of both executions
+  being merged into the steps.
+- **A heal whose assertion does not hold as written says so on that
+  step.** The failed step shows the guard's "Assertion failed as written"
+  reason instead of what replay missed first, and the execution trace
+  shows the steps the agent carried after it.
+- **A failed agent session no longer marks every step failed.** When a
+  heal did not succeed, the report and the replay showed all steps as
+  failed, including the ones replay had passed before the agent took
+  over. They now show what happened: passed up to the step replay stopped
+  on, that step failed with its reason, the rest skipped. When the agent
+  carried that step and failed on a later one, the later step is the
+  failed one, with the agent's reason. A failed recording shows the steps
+  the agent got through the same way.
+- `saffron trace` and `saffron report` keep working on a machine with no
+  browser opener (a container, an SSH or WSL remote): they print the
+  address to open instead of crashing.
+
+Editors (VS Code extension 0.3.0, JetBrains plugin 0.3.0)
+
+- **Open Replay** on a scenario in the Last run panel runs `saffron trace`
+  for it, by feature and scenario name, so two features with a scenario
+  of the same name are told apart. JetBrains also gets `trace` as a run
+  configuration command: its *Scenario* field takes a name or
+  `feature:scenario` as it is or wrapped whole in double quotes, and blank
+  opens every traced scenario, starting on a failed one.
+- **A Last run row stays with its run.** In VS Code, a row (its
+  screenshot, Open Replay, Open Scenario) acts on the project it came
+  from, even after the panel has moved to another one, and says so if
+  that project is no longer open.
+  In both editors, a screenshot is checked against the digest its run
+  recorded when you open it, so a picture a later run put at the same
+  path is refused instead of shown as this run's, and what opens is a copy
+  of the picture that was checked, named after the scenario. VS Code keeps
+  the copies in the extension's own storage, one folder per window,
+  removed when the window closes; JetBrains in the IDE's system folder,
+  removed after a day. Neither uses the shared temporary folder. In
+  JetBrains, double-clicking such a row no longer jumps to the scenario
+  over the message saying why. In VS Code, Open Scenario on a row whose
+  feature file was renamed or deleted since the run says so.
+  `saffron status --json` now reports each picture's `sha256`.
+- **Accept and reject decide the proposal you saw.** Both Proposals
+  panels send the revision they listed, so a proposal a later run
+  replaced after the panel showed it is refused with a prompt to refresh,
+  not decided unseen. A replaced proposal comes back unticked in both.
+  With a runner that lists no revision, they send the file as before.
+  VS Code's Accept all tooltip now says what `--all` does: it accepts
+  what is pending when it runs, skipping UNVERIFIED proposals and
+  refusing STALE ones.
+
 ## 0.8.5
 
 Released 2026-09-24.

package/README.md

@@ -1,10 +1,10 @@
 # Saffron
 
-**Gherkin-native test runner with zero-token cached replay and runtime AI healing.**
+**End-to-end tests written in plain language, with zero-token cached replay and runtime AI healing.**
 
 Free to use · [npm: saffron-ai](https://www.npmjs.com/package/saffron-ai) · [site](https://saffron-ai.io)
 
-Write tests as Gherkin feature files: `.saffron`, Saffron's superset of Gherkin with step sets, or standard `.feature`. On the first run, an AI agent (Claude Agent SDK + Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**. A human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
+Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude Agent SDK + Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**. A human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
 
 When the UI changes and a cached step fails at runtime, the agent takes over mid-execution with full scenario context, adapts, and finishes the run, then files a **cache proposal** you accept or reject, like a snapshot test update. Reports state exactly what was adapted and what it cost.
 
@@ -22,7 +22,7 @@ When the UI changes and a cached step fails at runtime, the agent takes over mid
 
 Like a maturing Cucumber suite, most of a new feature file is steps you already have. Saffron derives a **step index** from committed caches and, when recording a new scenario, replays every seeded island of known steps zero-AI. The agent records only the genuinely novel steps, wherever they sit. Exact step text is the identity (Cucumber-style); quoted values may differ (`"admin"` seeds from `"bob"`); data-table steps seed on matching keys. Measured on a live site: a 6-step scenario with one novel step recorded for **$0.15 / 6 AI calls**, vs $2.07 for a full unseeded recording of comparable length.
 
-**Step sets** (shipped): `.saffron` files, a superset dialect of Gherkin, add the `StepSet:` keyword for named, reusable step sequences invoked with `StepSet <name>` inside any scenario. Sets expand at parse time, so their steps cache and seed like ordinary steps; editing a set makes every invoking scenario honestly stale (re-recorded mostly seeded), and heal edits route to the set definition, one fix, every caller follows. Sets are **project-wide**: keep application-wide flows in a sets-only library file (convention: `features/shared.steps.saffron`) and invoke them from any feature.
+**Step sets** (shipped): `.saffron` files add the `StepSet:` keyword for named, reusable step sequences invoked with `StepSet <name>` inside any scenario. Sets expand at parse time, so their steps cache and seed like ordinary steps; editing a set makes every invoking scenario honestly stale (re-recorded mostly seeded), and heal edits route to the set definition, one fix, every caller follows. Sets are **project-wide**: keep application-wide flows in a sets-only library file (convention: `features/shared.steps.saffron`) and invoke them from any feature.
 
 **IDE integration & authoring** (shipped): `saffron steps` lists the project vocabulary with recorded/divergent/unrecorded badges (`--snippets` for native VS Code completion), `saffron mcp` serves it to AI assistants, `saffron lsp` brings completion/navigation/diagnostics to **JetBrains** (incl. Community editions via LSP4IJ) and Neovim, `saffron author` drafts feature files from prose in your own vocabulary, and duplicate wordings that record identical actions get **behavior-proven rename proposals**. The **Saffron VS Code extension** is on the [Marketplace](https://marketplace.visualstudio.com/items?itemName=ChathurangaJayasinghe.saffron-vscode) (`code --install-extension ChathurangaJayasinghe.saffron-vscode`). JetBrains users get the same in one click from the **Saffron JetBrains plugin** ([Marketplace](https://plugins.jetbrains.com/plugin/34240-saffron), [source](https://github.com/s-chathuranga-j/saffron-jetbrains-plugin)): `.saffron` file type, bundled grammar, LSP4IJ wiring. Both editors also **run from the IDE**: right-click Run on `.saffron` files and folders, Saffron run configurations (JetBrains), and a Saffron panel with feature files, tags to run, proposals to accept or reject, vocabulary health, config, and the run report as an in-editor dashboard, all fed by `saffron status --json`.
 
@@ -72,7 +72,7 @@ node dist/cli/index.js -p examples accept --all
 
 | Command | What it does |
 |---|---|
-| `saffron run [paths] [--headed] [--filter @tags] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`. Exit 1 on red (and on yellow with `--strict`). |
+| `saffron run [paths] [--headed] [--filter @tags] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
 | `saffron accept [files... \| --all] [--include-unverified] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
 | `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config. |
@@ -82,6 +82,7 @@ node dist/cli/index.js -p examples accept --all
 | `saffron lsp` | Run the Saffron language server over stdio for JetBrains (LSP4IJ/Ultimate), Neovim, any LSP editor: badge completion, StepSet go-to-definition, hover, diagnostics. Highlighting for JetBrains comes from the bundled TextMate grammar (`textmate/saffron` in the package); run `npx saffron lsp` in a terminal for setup steps. |
 | `saffron init [--agents list] [--examples]` | Make the project agent-ready (`--examples` installs the Saucedemo example suite): installs the bundled **Saffron agent skill** into `.claude/skills` and `.agents/skills` (Copilot/Cursor via `--agents`), registers the `saffron mcp` server in `.mcp.json` / `.cursor/mcp.json` / `.vscode/mcp.json`, adds a managed block to `AGENTS.md`/`CLAUDE.md`, scaffolds config. |
 | `saffron report` | Open the latest HTML report. |
+| `saffron trace [scenario] [--port n] [--no-open] [--json]` | Open the execution replay of the last run (every traced scenario, or the one you name): its Given / When / Then, the page inspectable at every action, the network alongside, and what the agent changed. Needs `"trace": "retain-on-failure"` (or `"on"`). |
 
 Configuration lives in `saffron.config.json` at your project root:
 
@@ -130,7 +131,7 @@ The full execution-model diagram is on the site: https://saffron-ai.io/docs
   proposals/     pending AI-generated cache updates (review these)
   history.jsonl  one line per run: trends (commit recommended)
   reports/       latest.html / latest.json (gitignore these)
-  artifacts/     failure screenshots from the latest run (gitignore these)
+  artifacts/     screenshots and traces from the latest run (gitignore these)
 features/        your .saffron and .feature files
 ```