saffron-ai 0.9.2 → 0.9.4

package/CHANGELOG.md

@@ -2,9 +2,199 @@
 
 All notable changes to Saffron, in one place: the `saffron-ai` runner (npm),
 the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
-JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates.
+JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
+release heading names its main feature: `## <version>: <title>`.
 
-## 0.9.2
+## 0.9.4: The suite's quality at a glance
+
+Released 2026-10-04.
+
+Runner
+
+- **The agent skill knows the new features.** It shows tag expressions
+  in `--filter`, `saffron trace`, and a CI job that exports its heals as
+  a bundle (or a draft pull request) for `saffron runs` and `saffron
+  import --run`. The AGENTS.md / CLAUDE.md block `saffron init` writes
+  says how a CI heal arrives, and the `project_status` MCP tool says
+  which run an imported proposal came from. Run `npx saffron init` again
+  to update a project's copy.
+- **`saffron dashboard`: the suite's quality at a glance.** A page in
+  the run report's design, read from `.saffron/history.jsonl`: tests,
+  the share passing (each scenario's latest result), failing scenarios
+  and for how many runs in a row, flaky ones (passed and failed by turns
+  over the last 20 runs), ones the agent healed again and again, never
+  run and left out of the last run, the slowest, pass rate and execution
+  time per run, and results per tag. It writes
+  `.saffron/reports/dashboard.html` and opens it; `--stdout` prints the
+  page for the IDE panels. The same numbers are in `saffron status
+  --json` as `quality`.
+
+VS Code extension 0.5.0
+
+- **The dashboard is the suite's quality page**, the one `saffron
+  dashboard` renders, instead of the run report. Click a scenario's
+  name to open it. Needs saffron-ai 0.9.4 or later.
+- **Last run: Open Replay and Open Report in the title bar.** Open
+  Replay opens every scenario the run traced, so it works when
+  everything passed and no row is listed; Open Report opens the HTML
+  report in the browser.
+
+JetBrains plugin 0.5.0
+
+- **The Dashboard is the suite's quality page**, the same one as in VS
+  Code, instead of the embedded run report. Click a scenario's name to
+  open it. Needs saffron-ai 0.9.4 or later.
+- **Last Run: Open Report** opens the run's HTML report in the browser.
+- **Last Run: Open Replay works when everything passed.** With no row to
+  select it did nothing; it now opens every scenario the run traced, or
+  says how to keep traces when the run kept none.
+
+## 0.9.3: Review CI heals on your machine
+
+Released 2026-10-04.
+
+Runner
+
+- **A heal made in CI can be reviewed and accepted on your machine.**
+  `saffron export` packs the pending proposals, the run's report and the
+  screenshots and traces it names into one zip, with where the run
+  happened (repository, branch, commit, CI run, shard); CI uploads it as
+  an artifact. `saffron import` checks it against your checkout and files
+  it where a local run would have, so `saffron diff`, `saffron accept`,
+  `saffron trace`, the report and both IDE panels work on it unchanged. A
+  bundle from another repository (by its path; the same path under
+  another host name, such as an SSH host alias, imports with a warning),
+  one that does not say its repository (unless
+  `--allow-unknown-repository`), of a newer format, with a damaged file or
+  a report it cannot show, or with a file over 1 GB (or over 100 times
+  its size unzipped) is refused whole, before anything is written, and so
+  are bundles that add up to over 8 GB unzipped together; a proposal made
+  against a different scenario text or committed recording is refused by
+  name, and the rest import. Bundles are read from disk one file at a
+  time, a URL's downloaded to disk first, so every shard of a large run
+  imports at once. Text from a bundle, in errors and reasons too, is
+  printed without control characters. Export keeps to the same limits:
+  it refuses a file over 1 GB and, when a bundle would be over 4 GB,
+  leaves the traces of green scenarios out first and says so. Shard
+  bundles merge into one report. The imported report says where it came
+  from and links to the CI run. Examples for GitHub Actions and Azure
+  Pipelines are in the guide (§19).
+- **Or CI opens a draft pull request with the heals.**
+  `saffron export --branch <name>` commits the proposal files, not the
+  recordings, onto the commit the run tested (on a pull request build, the
+  head of its branch, where each proposal's binding is checked again and
+  one that does not fit is left out, judged by that commit's own StepSets
+  and data) as a new branch, pushes it and, on GitHub, opens a
+  draft pull request through `gh` whose description is each proposal's
+  verification and diff, with the commands to accept, and a staging line
+  that stages exactly what accept changed (`git add -A .saffron/cache &&
+  git add -u .saffron/proposals`). After a tag build, which tested no
+  branch, name the pull request's branch with `--base`. Only the
+  checkout's HEAD, a commit it merged in or one it descends from is ever
+  built on; a `pull_request_target` build records the base branch and the
+  commit it checked out, never the fork's. The reviewer
+  switches to the branch, runs `saffron accept`, commits and marks it
+  ready: no agent session on their machine. The CI checkout is left as it
+  was, and an existing branch is never pushed onto.
+- **`saffron status --json` says which CI run a proposal was imported
+  from** (`importedFrom`: repository, branch, commit, run and its link,
+  shard), for as long as it is the proposal that was imported, so the
+  editors can show it past the next local run.
+- **`saffron runs` and `saffron import --run <id>`, for GitHub Actions and
+  Azure Pipelines alike.** `runs` lists the branch's CI runs (through `gh`,
+  or `az` with its azure-devops extension), with how many of each run's
+  proposals are pending here; `import --run` downloads a run's bundles and
+  imports them. The repository's host picks the service; a `ci` key in the
+  config sets it, and the Azure DevOps organization and project, for a
+  GitHub repository built by Azure Pipelines. An Azure pull request build
+  is listed under the branch it validates, and only the repository's own
+  builds are listed. Bundles of a re-run take each shard from its latest
+  attempt. `import --run` downloads into a folder under `.saffron/reports`
+  and removes it when done. A download that takes over 10 minutes is
+  stopped and says so. A tag build is listed under its tag on both
+  services. An Azure DevOps SSH remote through a host alias counts as
+  Azure. With `--json`, every `import` failure, an invalid config
+  included, is `{"error": ...}` and exit 2. A `runs --json` whose `gh` or
+  `az` is missing, signed out or failing answers `{provider, branch,
+  runs: [], problem: {kind, message}, error}`, `error` holding the same
+  message, so a client reading either field works; any other failure is
+  `{"error": ...}` alone; both exit 2. On Azure Pipelines the pull
+  request builds are asked for once (`--top 500`), so a listing usually takes
+  three `az` calls, not seven.
+  `export --branch` opens the draft pull request on Azure DevOps too,
+  through `az repos pr create`.
+- **Export streams the bundle to disk.** Screenshots and traces are
+  copied into the zip in chunks, never all held in memory, and the zip
+  is renamed into place only when complete; a file that changes while it
+  is packed is refused.
+- **Failures are one line.** `export` exits 2 for every failure that
+  pushed nothing (an `--out` that is a folder, an invalid config with
+  `--branch`), keeping exit 1 for a branch pushed without its pull
+  request. Any command's expected failure prints `saffron: <message>`
+  and exits 2, never a stack; `SAFFRON_DEBUG=1` shows it.
+- **Safe branch names in the pull request.** The description quotes the
+  branch in its commands, and `--branch` or `--base` with a space, a
+  quote or any of `` $ ` ; | & < > ( ) \ `` is refused up front.
+- **A `pull_request_review` build records its pull request's branch**,
+  read from the event, not `12/merge`.
+- **`import --json` passes on only the manifest fields it knows**, cleaned
+  of control characters.
+- **The pull request's review steps run saffron-ai by name**
+  (`npx -y -p saffron-ai saffron diff`): in a checkout without
+  `node_modules`, a bare `npx saffron` would fetch the unrelated `saffron`
+  package.
+- **A bundle names its run as the CI service does** (`github:<run id>`,
+  `azure:<build id>`), even with `SAFFRON_RUN_ID` set, so `saffron runs`
+  and both editors count its proposals under the right run.
+- **A green scenario's trace over 1 GB is left out of the bundle** like
+  the other green traces, instead of failing the export.
+- **Import hashes, compares and copies traces in chunks**: a 1 GB trace is
+  never held in memory. A bundle that changes on disk while it is
+  imported no longer says "nothing was imported" after some files were
+  written.
+- **`saffron runs` on Azure asks for 50 pull request builds first**, and
+  for more only while too few are the branch's, not 500 on every list.
+
+VS Code extension 0.4.0
+
+- **CI runs**: a new view lists the current branch's CI runs, GitHub
+  Actions or Azure Pipelines, through `saffron runs`. **Import** runs
+  `saffron import --run`, then lists under the run what happened to each
+  proposal and any warning (a checkout at another commit, a scenario that
+  changed since). Imported proposals are marked with the run they came
+  from, linked in the tooltip. From a red CI run to an accepted repair
+  without leaving the editor; commit and push with the Source Control
+  view. The CI's own tool keeps the credentials.
+- **Text from CI cannot act in a tooltip.** Run titles, branch names and
+  a bundle's notes are escaped so they never form links, headings or
+  other markup; a run links out only to GitHub or Azure DevOps, and never
+  when the address carries a user name. Import results are kept per
+  project. An import can be cancelled, and on Windows cancelling stops
+  the whole process tree. The list follows a branch switch, and makes no
+  CI call while its view is hidden.
+- **Cancelling an import stops all of it, and says what it left.** On
+  Windows the whole process tree is stopped however the CLI was started;
+  on macOS and Linux a process that outlives SIGTERM gets SIGKILL. The
+  message says a cancelled import may have written part of the run.
+
+JetBrains plugin 0.4.0
+
+- **CI Runs tab**: the same runs and the same Import, through the same
+  two commands, with each proposal's outcome under its run. The Proposals
+  tab names the CI run an imported proposal came from.
+- **Text from CI is shown as plain text**, in CI Runs and in Last Run
+  after an import, never as markup. Import is a background task you can
+  cancel, with no time limit; a run is imported once even if you switch
+  projects meanwhile; Refresh never starts a second list. The tab loads
+  as soon as the project is trusted, and its processes stop when the
+  project closes.
+- **Choosing another project clears the tabs at once**: a run listed for
+  the last project can no longer be imported into the new one, and a
+  status that arrives late for the last project is dropped.
+- **CI Runs lists again when you check out another branch.** A cancelled
+  import says it may have written part of the run.
+
+## 0.9.2: Bring your own AI agent
 
 Released 2026-09-30.
 
@@ -256,7 +446,7 @@ Editors (VS Code extension 0.3.3, JetBrains plugin 0.3.3)
   CSV; Run Tagged skips tags `--filter` cannot express and says which;
   diagnostics clear when a file closes or is deleted.
 
-## 0.9.1
+## 0.9.1: Tag expressions: and, or, not
 
 Released 2026-09-29.
 
@@ -320,7 +510,7 @@ Editors (VS Code extension 0.3.2, JetBrains plugin 0.3.2)
   tabs, under any file icon theme that has no icon of its own for them
   (Seti, the default, included).
 
-## 0.9.0
+## 0.9.0: Time-travel replay
 
 Released 2026-09-27.
 
@@ -515,7 +705,7 @@ Editors (VS Code extension 0.3.0, JetBrains plugin 0.3.0)
   what is pending when it runs, skipping UNVERIFIED proposals and
   refusing STALE ones.
 
-## 0.8.5
+## 0.8.5: Proposals bound to what you reviewed
 
 Released 2026-09-24.
 
@@ -554,7 +744,7 @@ Editors (VS Code extension 0.2.10, JetBrains plugin 0.2.5)
 - Both Proposals panels mark a stale proposal STALE, with the reason and
   what to do, and note a proposal filed by an older runner.
 
-## 0.8.4
+## 0.8.4: Heal and replay fixes
 
 Released 2026-09-24.
 
@@ -667,7 +857,7 @@ Editors (VS Code extension 0.2.9)
   files and to the config now refresh every open feature file, including a
   custom data folder and environment overlays.
 
-## 0.8.3
+## 0.8.3: Data references in table cells
 
 Released 2026-09-22.
 
@@ -683,7 +873,7 @@ Runner
   Code currently defaults to, which drifts) and which values it takes, so a
   project can pin its recording model.
 
-## 0.8.2
+## 0.8.2: Split a suite across CI machines
 
 Released 2026-09-22.
 
@@ -706,7 +896,7 @@ Runner
   twice does not count the run twice. Setup and teardown run on each shard
   and receive `SAFFRON_SHARD`. The docs have a GitHub Actions example.
 
-## 0.8.1
+## 0.8.1: Chrome, Edge, and setup and teardown
 
 Released 2026-09-22.
 
@@ -769,7 +959,7 @@ Editors (VS Code extension 0.2.8)
   hover and go-to-definition now follow the runner: the environment's CSV
   replaces the base one, and the hover names the file it read.
 
-## 0.8.0
+## 0.8.0: Test data outside the feature file
 
 Released 2026-09-21.
 
@@ -835,7 +1025,7 @@ Editors (VS Code extension 0.2.7, JetBrains plugin 0.2.4)
 - Both Orphans lists show the screenshot folders of scenarios that no longer
   exist, so "Remove All" never deletes something it did not show.
 
-## 0.7.2
+## 0.7.2: Failure screenshots
 
 Released 2026-09-21.
 
@@ -890,7 +1080,7 @@ Editors (JetBrains plugin 0.2.2)
   notification says so when it is missing. The presence check uses no
   plugin-manager API, so it stays clean as that API turns internal.
 
-## 0.7.1
+## 0.7.1: Truthful checks and safer edits
 
 Released 2026-09-20.
 
@@ -961,7 +1151,7 @@ Editors (VS Code extension 0.2.5)
   passes the tree item, not the file path the row click passes; both forms
   are accepted, and the project guard applies to both.
 
-## 0.7.0
+## 0.7.0: saffron diff, saffron prune, and an external review
 
 Released 2026-09-19.
 
@@ -1154,7 +1344,7 @@ Editors (JetBrains plugin 0.2.2, VS Code extension 0.2.4)
   longer shell syntax, and a file from a second workspace folder runs
   against that folder's config and install.
 
-## 0.6.0
+## 0.6.0: Multi-tab scenarios
 
 Released 2026-09-18.
 
@@ -1176,7 +1366,7 @@ Editors
   its short description fits without being cut off, and the overview
   introduces the tool and how to get started before listing features.
 
-## 0.5.7
+## 0.5.7: Claude Code plugin
 
 Released 2026-09-17.
 
@@ -1200,7 +1390,7 @@ Agents
   points at `saffron status`. Run `npx saffron init` again to refresh both
   in a project.
 
-## 0.5.6
+## 0.5.6: Fewer false duplicate wordings
 
 Released 2026-09-17.
 
@@ -1211,7 +1401,7 @@ Runner
   duplicate wordings; steps are compared by their quote-generalised
   pattern.
 
-## 0.5.5
+## 0.5.5: Duplicate wordings proven by behaviour
 
 Released 2026-09-17.
 
@@ -1223,7 +1413,7 @@ Runner
   "enter X in the username field" against "enter Y in the password
   field", which are different steps.
 
-## 0.5.4
+## 0.5.4: saffron status for the IDE panels
 
 Released 2026-09-17.
 
@@ -1243,7 +1433,7 @@ Editors
   Health sections and an in-editor dashboard. All read
   `saffron status --json`.
 
-## 0.5.3
+## 0.5.3: Cost by how you pay, and running from JetBrains
 
 Released 2026-09-17.
 
@@ -1284,7 +1474,7 @@ Editors
   extension, the last run's totals with the report one click away, and
   the proposals pending review with Accept all.
 
-## 0.5.2
+## 0.5.2: Scenario Outlines record correctly
 
 Released 2026-09-15.
 
@@ -1313,7 +1503,7 @@ Editors
   hover says where it is defined and never shows the value. In the VS
   Code extension and in `saffron lsp` for JetBrains and other editors.
 
-## 0.5.1
+## 0.5.1: The example suite in .saffron files
 
 Released 2026-09-14.
 
@@ -1330,7 +1520,7 @@ Editors
   parameters instead of the whole step line. Ships in the VS Code
   extension, the JetBrains plugin and the package's TextMate bundle.
 
-## 0.5.0
+## 0.5.0: Guided first run
 
 Released 2026-09-14.
 
@@ -1343,7 +1533,7 @@ Runner
   matching `saffron.config.json` with `retries: 1`. Only installs into an
   empty `features/` directory. Documented as "Your first run".
 
-## 0.4.2
+## 0.4.2: Recover from a wrong recording
 
 Released 2026-09-13.
 
@@ -1375,7 +1565,7 @@ Docs and site
 - Assertion policy documented on the Features page; cache lifecycle gained
   the staleness and wrong-recording stages.
 
-## 0.4.1
+## 0.4.1: JetBrains plugin
 
 Released 2026-09-13.
 
@@ -1397,7 +1587,7 @@ Editors
 - VS Code extension 0.1.3: plain punctuation in README, hover and
   diagnostic messages.
 
-## 0.4.0
+## 0.4.0: Agent skill for AI assistants
 
 Released 2026-09-13.
 
@@ -1423,13 +1613,13 @@ Editors
 - VS Code extension 0.1.2: warning and quick fix when a set name is written
   without the `StepSet` keyword.
 
-## 0.3.1
+## 0.3.1: Version fix
 
 Released 2026-08-15.
 
 - `saffron --version` no longer reports a hardcoded value.
 
-## 0.3.0
+## 0.3.0: Network-aware steps
 
 Released 2026-08-12.
 
@@ -1451,7 +1641,7 @@ Editors
 
 - VS Code extension 0.1.0 published to the Marketplace.
 
-## 0.2.0
+## 0.2.0: Secrets, cross-browser and parallel replay
 
 Released 2026-07-19.
 
@@ -1467,7 +1657,7 @@ Runner
   workers (`--workers N`), and a cheaper model for heal sessions
   (`--heal-model`).
 
-## 0.1.0
+## 0.1.0: First public release
 
 Released 2026-07-10.
 

package/README.md

@@ -76,13 +76,15 @@ node dist/cli/index.js -p examples accept --all
 | `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
 | `saffron accept [files... \| --all] [--include-unverified] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
-| `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config. |
+| `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
+| `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config, and suite quality from the run history (failing, flaky, never run, slowest, trends, per tag). |
 | `saffron steps [search] [--json] [--snippets]` | List/search the step vocabulary (files + caches) with recorded/divergent/unrecorded badges and usage; `--snippets` writes `.vscode/saffron.code-snippets` for native VS Code completion. |
 | `saffron mcp` | Serve the vocabulary and the project overview to AI assistants over stdio MCP (`search_steps`, `list_step_sets`, `project_status`), e.g. `claude mcp add saffron -- npx saffron mcp`. |
 | `saffron author <prose-file> [-o out]` | Draft a `.saffron` feature file from plain-paragraph requirements, reusing the project's step vocabulary (AI; reports how many lines are seedable). |
 | `saffron lsp` | Run the Saffron language server over stdio for JetBrains (LSP4IJ/Ultimate), Neovim, any LSP editor: badge completion, StepSet go-to-definition, hover, diagnostics. Highlighting for JetBrains comes from the bundled TextMate grammar (`textmate/saffron` in the package); run `npx saffron lsp` in a terminal for setup steps. |
 | `saffron init [--agents list] [--examples]` | Make the project agent-ready (`--examples` installs the Saucedemo example suite): installs the bundled **Saffron agent skill** into `.claude/skills` and `.agents/skills` (Copilot/Cursor via `--agents`), registers the `saffron mcp` server in `.mcp.json` / `.cursor/mcp.json` / `.vscode/mcp.json`, adds a managed block to `AGENTS.md`/`CLAUDE.md`, scaffolds config. |
 | `saffron report` | Open the latest HTML report. |
+| `saffron dashboard` | Open the suite's quality dashboard: failing, flaky and never-run scenarios, repeated heals, slowest, trends, per tag. |
 | `saffron trace [scenario] [--port n] [--no-open] [--json]` | Open the execution replay of the last run (every traced scenario, or the one you name): its Given / When / Then, the page inspectable at every action, the network alongside, and what the agent changed. Needs `"trace": "retain-on-failure"` (or `"on"`). |
 
 Configuration lives in `saffron.config.json` at your project root: