saffron-ai 0.9.4 → 0.9.6

package/CHANGELOG.md

@@ -5,6 +5,143 @@ the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates. Each
 release heading names its main feature: `## <version>: <title>`.
 
+## 0.9.6: Dashboard figures you can trust
+
+Released 2026-10-05.
+
+Runner
+
+- **Three dashboard figures no longer flatter the suite.** *Recorded*
+  counted a stale recording, one whose scenario was edited since and
+  needs the agent again, so the dashboard could say 100% while `saffron
+  status` said otherwise; it now counts current recordings only, and at
+  100% says "every scenario has a current recording". *Passing* and the
+  last run's pass rate counted a heal whose zero-AI proof had failed;
+  such a heal is now "healed, unverified", out of the pass rate, with its
+  own stripe in the health bar, a count on the tiles and a mark in the
+  healed-again list. *Flaky* counted a scenario that was red on one base
+  URL and green on another as flipping; flips are now counted between
+  consecutive runs on the same base URL, and a first recording that
+  failed is left out. `status --json` gains `quality.suite.unverified`,
+  `quality.lastRun.unverified`, and `unverified` on tags and healing
+  entries. Both IDE dashboards show the runner's page, so they follow
+  without a plugin update.
+- **An upload whose file does not exist says so.** Under a project path
+  reached through a link (a temp folder on macOS), the message was
+  "outside the project". The repository's own example now names its
+  upload file relative to the project, as replay has resolved it since
+  0.9.4.
+- **Uploads are single files.** A folder passed to a file input was let
+  through, and a folder upload sends everything inside it, dotfiles such
+  as `.env` included. The agent's upload and replay both refuse a folder
+  now.
+- **`saffron report --merge` never hides another environment's result.**
+  A re-run attempt replaced the earlier attempt of the same shard number
+  before checking that both ran the same suite against the same base
+  URL, so a green production attempt could silently drop a red staging
+  one. An attempt now replaces only the same shard of the same suite,
+  split and base URL; anything else is refused, naming the base URLs.
+- **`saffron export --branch` works with `"features": "."`.** With the
+  scenarios at the project root, the shared StepSets were left out of the
+  check against the commit the branch starts on, so a correctly bound
+  proposal was refused as using a StepSet "not defined anywhere".
+- A replay that finds its recording stale no longer leaves a listener on
+  the browser context.
+
+VS Code extension 0.5.1
+
+- **The dashboard tab shows the Saffron mark in dark themes.** It used the
+  activity bar icon, one dark shape that VS Code tints there and a tab
+  draws as is, so it was dark on a dark tab.
+- **Cancelling an import stops every process it started, on macOS and
+  Linux too.** The CLI runs in a process group of its own, so a `gh` or
+  `az` download that outlives it is stopped as well, with a forced stop
+  for one that ignores the first request. Output over 16 MB ends the
+  command with a clear message.
+
+## 0.9.5: CI heals you can trust
+
+Released 2026-10-04.
+
+Runner
+
+- **A recording can only use the environment variables its scenario
+  names.** Replay fails any other `{env:...}` by name, so a recording
+  someone commits, or a CI bundle you import, cannot read the rest of
+  your environment. The run's preflight checks the same set, masking
+  covers exactly what can resolve, and a `{env:...}` or `{data:...}`
+  the agent typed as plain text is recorded escaped and never resolves.
+  An outline whose Examples rows hold different `{env:...}` values now
+  replays each row with its own variable.
+- **Replay opens only http(s) pages and uploads only project files.**
+  Recorded navigation and new tabs refuse `data:` URLs, other `about:`
+  pages and other schemes, at replay and during recording. Upload paths
+  resolve against the project root, wherever saffron was started from.
+- **Proposals that came from elsewhere are not run with your secrets.**
+  A run replays a pending proposal at zero AI only when this machine
+  filed it. One imported from a CI run, or checked out from a heal
+  branch, is never replayed, healed over or recorded over: the scenario
+  reports yellow, pending review, with how to review it and the CI run
+  it came from. `--strict` fails on it like on any yellow; `saffron run
+  --rerecord` records over it. A CI build of a heal branch therefore
+  reports yellow until the proposals are accepted, and does not heal
+  again. After upgrading, a proposal filed by an earlier version is
+  reported as pending review on its first run: accept or reject it.
+- **Heal once.** When a committed recording fails and a heal filed on
+  this machine is still pending, the run replays that heal before
+  calling the agent. If it passes, the scenario is yellow, covered by
+  the pending proposal, so the next run and the outline's other Examples
+  rows do not pay for another heal. This works under `--no-agent` too.
+- **`saffron import --run` asks the CI service who started the run.** A
+  run started by a fork's pull request is refused unless you pass
+  `--allow-fork`, and a downloaded bundle that names a different run
+  from the one you asked for is refused.
+- **`saffron accept --all` takes only what you reviewed one by one.** It
+  skips proposals imported from a CI run unless you pass
+  `--include-imported`, and treats a proposal no proof replay checked as
+  unverified (`--include-unverified` to take it). Proposal files you name
+  are accepted as before. The Accept All buttons in both IDEs run this
+  command, so they are covered too.
+- **Re-runs and shards.** Import takes the latest attempt of a whole run
+  exported more than once, reports a conflict when shards of one run
+  filed different proposals for the same scenario, and refuses two
+  bundles holding the same shard. `--shard-by scenario` keeps all rows
+  of a Scenario Outline on one shard. Shard reports record the CI
+  attempt, and `saffron report --merge` merges the latest attempt of a
+  re-run shard.
+- **Feature edits are shown before they are applied.** `saffron diff`
+  (text and `--json`) and the draft pull request list the feature file
+  edits `saffron accept --with-feature-edit` would apply, and mark an
+  edit to a step set's step, which changes every scenario using it.
+  Feature edits can no longer hold line breaks or control characters,
+  accept skips one that would make the scenario name a new environment
+  variable, and every reader refuses a recording that names a feature
+  file outside the project.
+- **Text from a proposal, report or trace is printed as text.** The HTML
+  report and the `saffron trace` replay page escape every value, the
+  replay page runs under a Content-Security-Policy with sandboxed
+  snapshot frames, and `saffron diff`, `saffron accept` and the pending
+  listing strip terminal control sequences. `saffron import` refuses a
+  bundle whose zip holds a file twice, or whose report has a step status
+  or counts no run writes.
+- **Reports name proposal files relative to the project**, so a report
+  read on another checkout points at the right file. Older reports still
+  work.
+- Messages use plain punctuation, the missing `{env:VAR}` error
+  included.
+- **The dashboard's Failing count no longer forgets an old failure.** It
+  counted only failures from the last 10 runs, so a scenario that failed
+  and was then left out of ten runs read "all clear" while the health
+  bar still showed it red. It now counts each scenario's latest result,
+  as the health bar does.
+
+JetBrains plugin 0.5.1
+
+- **The Dashboard never shows another project's page.** A page still
+  loading when you switched projects (or when the tab was hidden) could
+  land afterwards and stay on screen; it is now dropped. Clicking a
+  scenario opens it in the project the page was made for.
+
 ## 0.9.4: The suite's quality at a glance
 
 Released 2026-10-04.
@@ -118,9 +255,9 @@ Runner
   `az` is missing, signed out or failing answers `{provider, branch,
   runs: [], problem: {kind, message}, error}`, `error` holding the same
   message, so a client reading either field works; any other failure is
-  `{"error": ...}` alone; both exit 2. On Azure Pipelines the pull
-  request builds are asked for once (`--top 500`), so a listing usually takes
-  three `az` calls, not seven.
+  `{"error": ...}` alone; both exit 2. On Azure Pipelines each listing
+  asks for 50 builds first, and for more, up to 500, only while too few
+  are the branch's.
   `export --branch` opens the draft pull request on Azure DevOps too,
   through `az repos pr create`.
 - **Export streams the bundle to disk.** Screenshots and traces are

package/README.md

@@ -74,7 +74,7 @@ node dist/cli/index.js -p examples accept --all
 | Command | What it does |
 |---|---|
 | `saffron run [paths] [--headed] [--filter expr] [--rerecord] [--no-agent] [--strict] [--browser b] [--no-hooks] [--workers n] [--heal-model m] [--no-verify] [--no-reuse] [--model m] [--storage-state f] [--trace mode]` | Run features. Cached replays are deterministic; misses/failures escalate to the agent (unless `--no-agent`). Replay cross-browser with `--browser firefox\|webkit`, parallelize with `--workers N`, heal on a cheaper model with `--heal-model`, keep an execution trace with `--trace retain-on-failure`. Exit 1 on red (and on yellow with `--strict`). |
-| `saffron accept [files... \| --all] [--include-unverified] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
+| `saffron accept [files... \| --all] [--include-unverified] [--include-imported] [--with-feature-edit] [--propagate]` | Promote cache proposals to committed caches (`--all` skips UNVERIFIED ones unless `--include-unverified`, and ones imported from a CI run unless `--include-imported`); `--with-feature-edit` also rewrites the adapted steps in the `.feature` file (and keeps the cache in sync); `--propagate` applies the heal's locator fixes to every other cache using the same locator. One heal repairs N scenarios before they ever fail. No args: list pending proposals. |
 | `saffron reject [files... \| --all]` | Discard proposals; the agent will try again next run. |
 | `saffron export [-o file] [--branch name]` / `saffron runs` / `saffron import <bundles...> \| --run <id>` | Carry a CI heal to your machine: export packs the run's proposals, report, screenshots and traces into one bundle for CI to upload; `runs` lists the branch's CI runs (GitHub Actions or Azure Pipelines); import (of a file, or of a run with `--run`) checks the bundle against your checkout (repository, each proposal's binding) and files it there, ready for `saffron diff` and `saffron accept`. `--branch` also has CI commit the proposals to a new branch and open a draft pull request with their diffs. |
 | `saffron status [--json]` | Project overview for people, IDE panels and dashboards: feature files and scenarios with cache state, tags, pending proposals, last run, history, vocabulary health (divergent steps, duplicate wordings proven by identical recordings), effective config, and suite quality from the run history (failing, flaky, never run, slowest, trends, per tag). |
@@ -120,7 +120,7 @@ experimental; `npx saffron login antigravity` once) or `cursor`
 (`npm i -D @cursor/sdk`, then `npx saffron login cursor`). `model` takes
 that provider's model names. `npx saffron login` says what a provider
 still needs on this machine. Every sign-in flow is in
-[docs/PROVIDERS.md](docs/PROVIDERS.md).
+[AI providers and sign-in](https://saffron-ai.io/docs/ai-providers).
 
 `storageState` (also `--storage-state`) points to a Playwright storage-state
 JSON (cookies + localStorage) applied to every scenario's browser context, so
@@ -150,13 +150,15 @@ features/        your .saffron and .feature files
 
 ## Roadmap
 
-See the roadmap on **[the site](https://saffron-ai.io/#roadmap)**, the single source of truth for
-milestones, statuses, and decision gates. Headlines:
-action-vocabulary additions as real-world failures surface them, the
-Level-2 shared step library (gated on Level-1 divergence data), LLM-less
-MCP replay, multi-provider agents, parallel workers, public npm/GitHub
-release, and the long-term open-core SaaS layer. Everything shipped so
-far is in the roadmap's "Where we are" table.
+The roadmap is at **[saffron-ai.io/docs/roadmap](https://saffron-ai.io/docs/roadmap)**:
+milestones, statuses and decision gates, and everything shipped so far in
+its "Where we are" table.
+Headlines ahead: API testing (reusable REST operations, fixtures and schema
+validation), scenarios with two users in two browsers, a configurable
+features folder, running one scenario (`file:line`, with run icons in both
+IDEs), parallel recording and a browser matrix in one run, failure video,
+agent-blind secrets, LLM-less MCP replay, the Level-2 shared step library
+(gated on Level-1 divergence data), and the long-term open-core SaaS layer.
 
 ## Documentation