saffron-ai 0.7.2 → 0.8.1

package/CHANGELOG.md

@@ -4,6 +4,135 @@ All notable changes to Saffron, in one place: the `saffron-ai` runner (npm),
 the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates.
 
+## 0.8.1
+
+Released 2026-09-21.
+
+Runner
+
+- **Google Chrome and Microsoft Edge: `--browser chrome` and
+  `--browser msedge`.** Run on the branded browser your users actually
+  have, not only the open-source Chromium build. Both speak CDP, so
+  everything works there: replay, recording and healing. Recordings are the
+  same files, so a scenario recorded on `chromium` replays on `chrome` and
+  `msedge` and the other way round. Saffron uses the copy installed on the
+  machine; when it is missing the run stops before the first scenario, exit
+  code 2, and names the browser (`npx playwright install chrome` adds it on
+  a CI image). Also accepted as `browser` in the config.
+- **Setup and teardown: `"setup"` and `"teardown"` in the config.** Shell
+  commands, one or a list, that Saffron runs once around a run, in the
+  project root: seed the data your scenarios read before any browser
+  opens, and clean up after the last scenario. A failed setup stops the
+  run with exit code 2 and no scenario runs. Teardown always runs: after
+  failures, after a failed setup, after Ctrl-C (which first stops a running
+  setup and waits for it, so nothing writes behind the cleanup). A failed teardown leaves
+  the verdicts alone and turns an otherwise green exit into 2, because what
+  it left behind is the next run's false red. The commands receive
+  `SAFFRON_BASE_URL` and `SAFFRON_ENV`. `--no-hooks` skips them,
+  `hookTimeoutMs` (default five minutes) bounds each one, and the report
+  lists every command with how it ended. With `{unique:name}` from 0.8.0
+  this makes a suite re-runnable and parallel-safe.
+
+Fixes
+
+- **Step reuse could bring back a stale data value.** When a `{data:...}`
+  value changed, the scenario's recording correctly went stale, but a new
+  recording could be seeded from that same recording's steps, putting the
+  old literal back under a fresh data snapshot, verified, at zero AI calls.
+  A step whose source recording still spells out a value that has since
+  changed is no longer reused.
+- **`every {data:list}` counted hidden content as shown.** A value sitting
+  in a `hidden` or `display: none` element satisfied the check. The target
+  must now be visible, and only text the page actually shows counts. A
+  native `<select>` keeps its exception, since its options are never
+  visible while it is closed, but only when the select itself is visible.
+  Text is still compared as written, so CSS `text-transform` does not
+  change the result.
+- **`{unique:name}` inside an Examples cell, a table cell or a doc string
+  reached the browser as the literal token.** Unique values were resolved
+  before parameters were substituted. They are now resolved in parameter
+  values too, with the same per-run seed, for values and locators alike.
+- **`saffron prune` could list the screenshots of a live scenario as
+  orphaned** when its feature file is outside the configured directory and
+  the scenario has only ever failed, so no cache or proposal named the
+  file. Each screenshot folder now records which feature and scenario it
+  belongs to, prune reads it before judging, and the listing names what
+  the pictures were of.
+
+Editors (VS Code extension 0.2.8)
+
+- **VS Code ignored environment-specific CSV files.** With environment
+  `staging` and `roles.staging.csv`, `Examples: {data:roles}` was reported
+  as not resolving, or described from the base `roles.csv`. Diagnostics,
+  hover and go-to-definition now follow the runner: the environment's CSV
+  replaces the base one, and the hover names the file it read.
+
+## 0.8.0
+
+Released 2026-09-21.
+
+Test data moves out of the feature file: a `data/` folder, `{data:...}`
+references, list checks, `Examples` from a file, values unique per run, and
+data per environment.
+
+Runner
+
+- **Test data outside the feature file: `{data:file.key}`.** Put values in
+  JSON files under `data/` and reference them from a step, a table cell or
+  a doc string: `When I sign in as {data:users.admin.email}`. Same family
+  as `{env:VAR}`: the recording stores the token and replay reads the file,
+  so changing a value changes the next replay at zero tokens. A recording
+  that still spells out the old value goes stale instead of replaying
+  yesterday's data, and a missing file or key stops the run by name before
+  a browser opens. New config key `dataDir` (default `data`). Not for
+  secrets: data files are committed.
+- **Check a whole list in one step.** `Then the status filter should list
+  every {data:enums.OrderStatus}` records ONE assertion (`expectEach`) that
+  names the list and its container. Replay reads the list from the file, so
+  a value added to the data is checked on the next run at zero tokens, and
+  a missing one fails by name with how many were found. Values match as
+  whole text ("Paid" is not "Unpaid"), extra entries such as "All" are
+  allowed, and a closed dropdown works.
+- **`Examples` from a file**, in `.saffron` files: `Examples: {data:roles}`
+  reads `data/roles.csv` (header row = placeholders) or a JSON list of
+  records. One recording, one zero-token replay per row; a row added to the
+  file records nothing. A placeholder without a column, a missing file and a
+  `.feature` file using it are each parse errors that say what to do.
+- **`{unique:name}`**: a value that is new on every run and the same within
+  one, for scenarios that create things. The agent types the generated
+  value while recording and Saffron turns it back into the token; a run that
+  is not green lists what each token stood for. `{unique:name:digits}` for
+  numeric fields.
+- **Data per environment**: `--env staging` (or `SAFFRON_ENV`, or `env` in
+  the config) lays `data/users.staging.json` over `data/users.json`, key by
+  key. The reference in the step stays the same on every target.
+- `saffron status --json` gains `data`: the folder, the environment, the
+  files, and references that do not resolve. `saffron lsp` completes
+  `{data:` paths, shows values on hover, jumps to the key and flags
+  unresolved references. The bundled grammar highlights `{data:...}` and
+  `{unique:...}`. `saffron init --examples` installs `data/catalog.json` and
+  a scenario that uses it, and the agent skill, its syntax reference and the
+  AGENTS.md block teach the new syntax.
+
+- **`saffron prune` covers failure screenshots.** A deleted scenario also
+  leaves its screenshot folder behind under `.saffron/artifacts/`. Prune
+  lists those folders with the caches and proposals and removes them with
+  `--yes`; a live scenario's folder is kept, per Examples row. `saffron
+  status --json` reports them as orphans of kind `artifact`, and both IDE
+  panels (VS Code 0.2.7, JetBrains 0.2.4) show them.
+
+Editors (VS Code extension 0.2.7, JetBrains plugin 0.2.4)
+
+- **`{data:...}` in the editor.** VS Code completes data files and keys with
+  their values, shows the current value (or a list's values, or an Examples
+  file's columns) on hover, jumps to the key, and underlines a reference
+  that does not resolve, reading the same files and environment overlay the
+  runner does. JetBrains gets the same through `saffron lsp`. Both highlight
+  `{data:...}` and `{unique:...}` like `{env:VAR}`, including on an
+  `Examples:` line.
+- Both Orphans lists show the screenshot folders of scenarios that no longer
+  exist, so "Remove All" never deletes something it did not show.
+
 ## 0.7.2
 
 Released 2026-09-21.

package/README.md

@@ -72,7 +72,7 @@ node dist/cli/index.js -p examples accept --all
 
 | Command | What it does |
 |---|---|
-| `saffron run [paths] [--headed] [--filter @tags] [--rerecord] [--no-agent] [--strict] [--browser b] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`. Exit 1 on red (and on yellow with `--strict`). |
+| `saffron run [paths] [--headed] [--filter @tags] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`. Exit 1 on red (and on yellow with `--strict`). |
 | `saffron accept [files... \| --all] [--include-unverified] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
 | `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config. |
@@ -89,6 +89,7 @@ Configuration lives in `saffron.config.json` at your project root:
 {
   "baseURL": "http://localhost:4173",
   "features": "features",
+  "dataDir": "data",
   "actionTimeoutMs": 5000,
   "pollIntervalMs": 100,
   "retries": 1,