saffron-ai 0.9.3 → 0.9.5

package/CHANGELOG.md

@@ -5,6 +5,133 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
 release heading names its main feature: `## <version>: <title>`.
 
+## 0.9.5: CI heals you can trust
+
+Released 2026-10-04.
+
+Runner
+
+- **A recording can only use the environment variables its scenario
+  names.** Replay fails any other `{env:...}` by name, so a recording
+  someone commits, or a CI bundle you import, cannot read the rest of
+  your environment. The run's preflight checks the same set, masking
+  covers exactly what can resolve, and a `{env:...}` or `{data:...}`
+  the agent typed as plain text is recorded escaped and never resolves.
+  An outline whose Examples rows hold different `{env:...}` values now
+  replays each row with its own variable.
+- **Replay opens only http(s) pages and uploads only project files.**
+  Recorded navigation and new tabs refuse `data:` URLs, other `about:`
+  pages and other schemes, at replay and during recording. Upload paths
+  resolve against the project root, wherever saffron was started from.
+- **Proposals that came from elsewhere are not run with your secrets.**
+  A run replays a pending proposal at zero AI only when this machine
+  filed it. One imported from a CI run, or checked out from a heal
+  branch, is never replayed, healed over or recorded over: the scenario
+  reports yellow, pending review, with how to review it and the CI run
+  it came from. `--strict` fails on it like on any yellow; `saffron run
+  --rerecord` records over it. A CI build of a heal branch therefore
+  reports yellow until the proposals are accepted, and does not heal
+  again. After upgrading, a proposal filed by an earlier version is
+  reported as pending review on its first run: accept or reject it.
+- **Heal once.** When a committed recording fails and a heal filed on
+  this machine is still pending, the run replays that heal before
+  calling the agent. If it passes, the scenario is yellow, covered by
+  the pending proposal, so the next run and the outline's other Examples
+  rows do not pay for another heal. This works under `--no-agent` too.
+- **`saffron import --run` asks the CI service who started the run.** A
+  run started by a fork's pull request is refused unless you pass
+  `--allow-fork`, and a downloaded bundle that names a different run
+  from the one you asked for is refused.
+- **`saffron accept --all` takes only what you reviewed one by one.** It
+  skips proposals imported from a CI run unless you pass
+  `--include-imported`, and treats a proposal no proof replay checked as
+  unverified (`--include-unverified` to take it). Proposal files you name
+  are accepted as before. The Accept All buttons in both IDEs run this
+  command, so they are covered too.
+- **Re-runs and shards.** Import takes the latest attempt of a whole run
+  exported more than once, reports a conflict when shards of one run
+  filed different proposals for the same scenario, and refuses two
+  bundles holding the same shard. `--shard-by scenario` keeps all rows
+  of a Scenario Outline on one shard. Shard reports record the CI
+  attempt, and `saffron report --merge` merges the latest attempt of a
+  re-run shard.
+- **Feature edits are shown before they are applied.** `saffron diff`
+  (text and `--json`) and the draft pull request list the feature file
+  edits `saffron accept --with-feature-edit` would apply, and mark an
+  edit to a step set's step, which changes every scenario using it.
+  Feature edits can no longer hold line breaks or control characters,
+  accept skips one that would make the scenario name a new environment
+  variable, and every reader refuses a recording that names a feature
+  file outside the project.
+- **Text from a proposal, report or trace is printed as text.** The HTML
+  report and the `saffron trace` replay page escape every value, the
+  replay page runs under a Content-Security-Policy with sandboxed
+  snapshot frames, and `saffron diff`, `saffron accept` and the pending
+  listing strip terminal control sequences. `saffron import` refuses a
+  bundle whose zip holds a file twice, or whose report has a step status
+  or counts no run writes.
+- **Reports name proposal files relative to the project**, so a report
+  read on another checkout points at the right file. Older reports still
+  work.
+- Messages use plain punctuation, the missing `{env:VAR}` error
+  included.
+- **The dashboard's Failing count no longer forgets an old failure.** It
+  counted only failures from the last 10 runs, so a scenario that failed
+  and was then left out of ten runs read "all clear" while the health
+  bar still showed it red. It now counts each scenario's latest result,
+  as the health bar does.
+
+JetBrains plugin 0.5.1
+
+- **The Dashboard never shows another project's page.** A page still
+  loading when you switched projects (or when the tab was hidden) could
+  land afterwards and stay on screen; it is now dropped. Clicking a
+  scenario opens it in the project the page was made for.
+
+## 0.9.4: The suite's quality at a glance
+
+Released 2026-10-04.
+
+Runner
+
+- **The agent skill knows the new features.** It shows tag expressions
+  in `--filter`, `saffron trace`, and a CI job that exports its heals as
+  a bundle (or a draft pull request) for `saffron runs` and `saffron
+  import --run`. The AGENTS.md / CLAUDE.md block `saffron init` writes
+  says how a CI heal arrives, and the `project_status` MCP tool says
+  which run an imported proposal came from. Run `npx saffron init` again
+  to update a project's copy.
+- **`saffron dashboard`: the suite's quality at a glance.** A page in
+  the run report's design, read from `.saffron/history.jsonl`: tests,
+  the share passing (each scenario's latest result), failing scenarios
+  and for how many runs in a row, flaky ones (passed and failed by turns
+  over the last 20 runs), ones the agent healed again and again, never
+  run and left out of the last run, the slowest, pass rate and execution
+  time per run, and results per tag. It writes
+  `.saffron/reports/dashboard.html` and opens it; `--stdout` prints the
+  page for the IDE panels. The same numbers are in `saffron status
+  --json` as `quality`.
+
+VS Code extension 0.5.0
+
+- **The dashboard is the suite's quality page**, the one `saffron
+  dashboard` renders, instead of the run report. Click a scenario's
+  name to open it. Needs saffron-ai 0.9.4 or later.
+- **Last run: Open Replay and Open Report in the title bar.** Open
+  Replay opens every scenario the run traced, so it works when
+  everything passed and no row is listed; Open Report opens the HTML
+  report in the browser.
+
+JetBrains plugin 0.5.0
+
+- **The Dashboard is the suite's quality page**, the same one as in VS
+  Code, instead of the embedded run report. Click a scenario's name to
+  open it. Needs saffron-ai 0.9.4 or later.
+- **Last Run: Open Report** opens the run's HTML report in the browser.
+- **Last Run: Open Replay works when everything passed.** With no row to
+  select it did nothing; it now opens every scenario the run traced, or
+  says how to keep traces when the run kept none.
+
 ## 0.9.3: Review CI heals on your machine
 
 Released 2026-10-04.
@@ -74,9 +201,9 @@ Runner
   `az` is missing, signed out or failing answers `{provider, branch,
   runs: [], problem: {kind, message}, error}`, `error` holding the same
   message, so a client reading either field works; any other failure is
-  `{"error": ...}` alone; both exit 2. On Azure Pipelines the pull
-  request builds are asked for once (`--top 500`), so a listing usually takes
-  three `az` calls, not seven.
+  `{"error": ...}` alone; both exit 2. On Azure Pipelines each listing
+  asks for 50 builds first, and for more, up to 500, only while too few
+  are the branch's.
   `export --branch` opens the draft pull request on Azure DevOps too,
   through `az repos pr create`.
 - **Export streams the bundle to disk.** Screenshots and traces are

package/README.md

@@ -74,16 +74,17 @@ node dist/cli/index.js -p examples accept --all
 | Command | What it does |
 |---|---|
 | `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
-| `saffron accept [files... \| --all] [--include-unverified] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
+| `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
 | `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
-| `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config. |
+| `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config, and suite quality from the run history (failing, flaky, never run, slowest, trends, per tag). |
 | `saffron steps [search] [--json] [--snippets]` | List/search the step vocabulary (files + caches) with recorded/divergent/unrecorded badges and usage; `--snippets` writes `.vscode/saffron.code-snippets` for native VS Code completion. |
 | `saffron mcp` | Serve the vocabulary and the project overview to AI assistants over stdio MCP (`search_steps`, `list_step_sets`, `project_status`), e.g. `claude mcp add saffron -- npx saffron mcp`. |
 | `saffron author <prose-file> [-o out]` | Draft a `.saffron` feature file from plain-paragraph requirements, reusing the project's step vocabulary (AI; reports how many lines are seedable). |
 | `saffron lsp` | Run the Saffron language server over stdio for JetBrains (LSP4IJ/Ultimate), Neovim, any LSP editor: badge completion, StepSet go-to-definition, hover, diagnostics. Highlighting for JetBrains comes from the bundled TextMate grammar (`textmate/saffron` in the package); run `npx saffron lsp` in a terminal for setup steps. |
 | `saffron init [--agents list] [--examples]` | Make the project agent-ready (`--examples` installs the Saucedemo example suite): installs the bundled **Saffron agent skill** into `.claude/skills` and `.agents/skills` (Copilot/Cursor via `--agents`), registers the `saffron mcp` server in `.mcp.json` / `.cursor/mcp.json` / `.vscode/mcp.json`, adds a managed block to `AGENTS.md`/`CLAUDE.md`, scaffolds config. |
 | `saffron report` | Open the latest HTML report. |
+| `saffron dashboard` | Open the suite's quality dashboard: failing, flaky and never-run scenarios, repeated heals, slowest, trends, per tag. |
 | `saffron trace [scenario] [--port n] [--no-open] [--json]` | Open the execution replay of the last run (every traced scenario, or the one you name): its Given / When / Then, the page inspectable at every action, the network alongside, and what the agent changed. Needs `"trace": "retain-on-failure"` (or `"on"`). |
 
 Configuration lives in `saffron.config.json` at your project root:
@@ -119,7 +120,7 @@ experimental; `npx saffron login antigravity` once) or `cursor`
 (`npm i -D @cursor/sdk`, then `npx saffron login cursor`). `model` takes
 that provider's model names. `npx saffron login` says what a provider
 still needs on this machine. Every sign-in flow is in
-[docs/PROVIDERS.md](docs/PROVIDERS.md).
+[AI providers and sign-in](https://saffron-ai.io/docs/ai-providers).
 
 `storageState` (also `--storage-state`) points to a Playwright storage-state
 JSON (cookies + localStorage) applied to every scenario's browser context, so
@@ -149,13 +150,15 @@ features/        your .saffron and .feature files
 
 ## Roadmap
 
-See the roadmap on **[the site](https://saffron-ai.io/#roadmap)**, the single source of truth for
-milestones, statuses, and decision gates. Headlines:
-action-vocabulary additions as real-world failures surface them, the
-Level-2 shared step library (gated on Level-1 divergence data), LLM-less
-MCP replay, multi-provider agents, parallel workers, public npm/GitHub
-release, and the long-term open-core SaaS layer. Everything shipped so
-far is in the roadmap's "Where we are" table.
+The roadmap is at **[saffron-ai.io/docs/roadmap](https://saffron-ai.io/docs/roadmap)**:
+milestones, statuses and decision gates, and everything shipped so far in
+its "Where we are" table.
+Headlines ahead: API testing (reusable REST operations, fixtures and schema
+validation), scenarios with two users in two browsers, a configurable
+features folder, running one scenario (`file:line`, with run icons in both
+IDEs), parallel recording and a browser matrix in one run, failure video,
+agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
+(gated on Level-1 divergence data), and the long-term open-core SaaS layer.
 
 ## Documentation