saffron-ai 0.9.1 → 0.9.2

package/CHANGELOG.md

@@ -4,6 +4,258 @@ All notable changes to Saffron, in one place: the `saffron-ai` runner (npm),
 the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates.
 
+## 0.9.2
+
+Released 2026-09-30.
+
+Runner
+
+- **Browser tool calls are checked call by call.** A security review of
+  the agent's tools found three ways a prompt-injected page could reach
+  the machine: `view-source:file://` slipped past Playwright's `file:`
+  check and read any file; the `filename` option of the snapshot,
+  screenshot, console and network tools wrote page content over any
+  project file (a response body into a bin is code execution on the next
+  run); and a file upload could hand `.env`, with every secret, to the
+  page. Navigation now opens web pages only, uploads take existing files
+  inside the project by their real path (a link out of the project is
+  out) and never dotfiles, `filename` is dropped, and anything the
+  vendor's tools write goes to a private folder of the session, removed
+  when it ends. The checks sit in the one
+  proxy every provider and both paths use, and replay applies the same
+  ones to a cache, which is data anyone can commit. A project reached
+  through a symlinked path (`/var` and `/private/var` on macOS) counts
+  as the project. Claude sessions also set
+  `strictMcpConfig`, so a `.mcp.json` or user-level MCP server of yours
+  never joins a recording. Codex sessions turn off its remaining
+  execution features (`unified_exec`, `multi_agent`, `code_mode_host`,
+  the in-app and external browsers) besides the shell.
+- **Go-to-definition works under a folder with a space, `#` or `%` in
+  its name, and on Windows.** The language server sent unencoded file
+  paths as URIs; a strict client rejected them. A nested project without
+  a `features` key now gets the default folder instead of the outer
+  project's, and on Windows a StepSet in a subfolder is no longer flagged
+  as defined twice.
+- **`--filter "not @wip @slow"` is refused** instead of reading as
+  `not @wip or @slow`: once `and`, `not` or parentheses appear, every
+  tag needs its `and` or `or`. The plain list forms are unchanged.
+- **Outline values no longer replace inside a test id or frame selector
+  by coincidence** (`add` in `add-to-cart`), while a short id with a
+  digit still does (`1234` in `order-1234`), and a `<name>` that opens a
+  regex named group is left alone at replay.
+- **The cost trend leaves a gap for runs whose provider reported no cost**
+  instead of plotting them as $0, and every other run keeps its place. Cursor's turn limit counts tool calls,
+  not the pieces their arguments stream in. An Antigravity session that
+  hits the turn limit says so, and how to raise it, even when agy
+  reports the session a success.
+
+- **No agent can run code on your machine through Playwright.**
+  Playwright MCP's `browser_run_code_unsafe` runs any JavaScript in the
+  Playwright server process (its own docs call it RCE-equivalent); a
+  Codex test session used it to read a file from the home directory. It
+  is now hidden from every provider's agent, Claude's included, and a call
+  to it by name is refused. Nothing recorded depends on it.
+  `browser_evaluate`, which runs in the page, stays.
+- **Claude recordings no longer carry your claude.ai connectors.** On a
+  Claude subscription login, Claude Code loads the account's claude.ai
+  connectors (Gmail, Figma, Canva, ElevenLabs and the like) into every
+  session from its second request on. On one account that added 294 tool
+  definitions, 1.16 MB of schema, to every model call: recording the
+  guestbook example cost $15.74 at API rates and 11.5M cache-read tokens.
+  Saffron's sessions now set `disableClaudeAiConnectors`, and the same
+  recording costs $0.37 with 279k cache-read tokens, in 70 s instead of
+  180 s. The agent never needed those tools; your connectors in Claude
+  itself are untouched.
+- **Record and heal on the subscription you already have.** Besides
+  Claude (still the default), `"provider"` in `saffron.config.json` (or
+  `--provider`) now takes `codex` (a ChatGPT plan, through
+  `@openai/codex-sdk`), `antigravity` (Google AI Pro/Ultra, through
+  Google's `agy` CLI; experimental) and `cursor` (through `@cursor/sdk`).
+  Each runs on the login its official client already has on the machine.
+  These agents reach the browser through a local MCP gateway that records
+  every call with the same hooks as Claude, so their recordings, heals and
+  proof replays work the same. `model` takes the provider's own model
+  names; `saffron author` takes `--provider` too. The two SDKs are
+  optional peer dependencies: install the one you use. Codex runs without
+  your own `~/.codex/config.toml` apart from its login settings (a keyring
+  login works), which needs macOS or Linux for now: on Windows the Codex
+  provider refuses to start rather than run with your own MCP servers.
+  `agy` gets an explicit `--print-timeout` sized to the turn budget, and
+  an `agy` that cannot start fails its scenario instead of the run.
+  Antigravity is verified live with agy 1.2.13 (record, proof replay and
+  heal): its MCP calls go through agy's `call_mcp_tool`, allowed by two
+  rules in agy's settings and limited by a hook to Saffron's two servers.
+- **`saffron login [provider]`** says what a provider still needs on this
+  machine, and does the one-time setup where one is needed:
+  `saffron login cursor` runs Cursor's sign-in, and
+  `saffron login antigravity` checks that agy is signed in and allows
+  Saffron's tools in agy's settings (two rules, everything else kept).
+- **New docs page: AI providers and sign-in** (`docs/PROVIDERS.md`), with
+  every provider's sign-in, what `saffron login` sets up, what Saffron does
+  in its sessions, how to undo it, and troubleshooting.
+- **A pass can no longer leave an assertion unchecked.** `saffron_done`
+  with success is refused once while an announced `Then` step has
+  recorded no check, and Saffron's tool refusals are now MCP errors, so an
+  agent that batches its calls stops and fixes the step instead of filing
+  an unverified proposal. A second pass stands (the proof replay still
+  marks it unverified), so no session burns its turns on it. A step
+  announced twice keeps one adaptation note, and an agent that fails
+  after `saffron_done` keeps its verdict.
+- **Node.js 20.6 or newer** is now required (22.13 for the Cursor
+  provider).
+- **Cost not reported, not $0.** Runs whose provider reports no price say
+  so in the summary line and the report instead of showing $0.0000.
+- **The editor finds a project nested inside a bigger folder.** When the
+  IDE is opened above the Saffron project (a solution folder with the
+  tests in a subfolder), `saffron lsp` now resolves each `.saffron` file
+  against the nearest `saffron.config.json` above it. StepSets defined in
+  a sibling file and `{data:...}` references were reported as undefined
+  in that layout; they resolve now, and completion, hover and
+  go-to-definition follow the same root.
+- **A browser tool result can no longer flood the agent's context.** A
+  recording fetched a 1.5 MB API response body with
+  `browser_network_request` (about 400k tokens). Claude Code then compacted
+  the session in a loop and failed the scenario after a few dollars of
+  tokens. Every browser tool result is now cut at 60,000 characters, with
+  a note asking for less (a snapshot of one region, a filtered request
+  list, or a `saffron_assert` on the field). The Playwright code Saffron
+  records comes first in a result, so a cut result still records its
+  action. The cap applies with and without the trace bridge, and to Codex,
+  Antigravity and Cursor sessions through the tool gateway.
+- **Each Outline row acts on its own element through a locator chain.** A
+  chain keeps the row's value inside its selector; that value now becomes
+  the row's `<param>` (or data-table reference) there too, so another row
+  no longer clicks the recorded row's card. The same goes for every other
+  field replay resolves: an assertion's regex pattern (an `href`, a
+  heading's name), a test id, a prompt answer, a frame and a drag's drop
+  target. An Outline checking a link's `href` failed as written on every
+  row but the recorded one.
+- **A changed table or data value still written in the recording is found
+  escaped too** (in a selector's quoted or regex body, or a pattern), so
+  the recording is made again instead of replaying the old value.
+- **Chains with a comma, an apostrophe or a role option record the element
+  they clicked.** `filter({ hasText: 'Paris, France' })` and
+  `getByText(/^Don't$/)` recorded nothing, a regex with an apostrophe made
+  a selector Playwright read differently, and `getByRole('heading', {
+  level: 2 })` recorded any heading. Role options are kept as selector
+  attributes; one with no selector form records nothing.
+- **Tag lists read the way 0.9.0 read them.** An empty entry (`@a,@b,`) is
+  skipped again, and a tag's own parentheses (`@issue(42)`) stay part of
+  it; a parenthesis after a space or a keyword still groups.
+- **Replay fixes.** A heal that fixes a network assertion is checked
+  against the response the agent's session produced, instead of failing
+  "as written" while waiting for a new one. A replay no longer inherits an
+  upload armed on its page by an earlier replay (a pending proposal that
+  failed), which failed its first click. A value resolved into a regex
+  pattern (`{env:VAR}`, `<param>`, `{data:...}`) matches as written: the
+  `+` in `a+b@x.com` or the `$` in `$9.99` no longer change the pattern.
+- **A failed tool call records nothing** in Codex, Antigravity and Cursor
+  sessions, as in Claude's: an upload refused because no file chooser was
+  open was recorded, and broke replay.
+- **Antigravity gets its prompt on stdin.** The prompt carries resolved
+  `{env:VAR}` values, and `agy -p` put it in the command line, readable by
+  every account on the machine.
+- **Tools a recording cannot replay are hidden from every agent:** going
+  back or forward, reloading, resizing and dropping files.
+  `browser_navigate` reaches any page instead, and is recorded.
+- **`saffron_done` is refused once when a `Then` step recorded only setup**
+  (a click that opens a panel) and no check, the rule replay applies.
+- **Playwright's MCP server gets the whole environment again** when the
+  trace bridge is off, so `PLAYWRIGHT_MCP_*` settings apply there too.
+- **A proposal lock held by a live process on this machine is broken after
+  an hour**: a pid reused after its holder died kept it forever.
+- **`saffron status` says "cost not reported"** for a provider with no
+  price, instead of $0.00. `saffron status --json` leaves such a cost
+  out (the last run's, history's and proposals'), and the editors'
+  last-run lines say "cost not reported" too (VS Code and JetBrains, next
+  editor release).
+- **A Codex or Cursor session stopped at the turn limit says so**, and to
+  raise `maxTurns`, instead of "ended without finalizing".
+- **`saffron login cursor` on Node below 22.13 names the Node version it
+  needs**, and a vendor SDK missing a part of its own reports that part
+  instead of asking to install the SDK again.
+- **A single locator with options records the element it acted on.**
+  `getByText('Save', { exact: true })`, `getByLabel('Email', { exact:
+  true })` and `getByText(/^Save$/)` recorded nothing, so the click or
+  fill was missing from the recording, and `getByRole('checkbox', {
+  checked: false })` recorded any checkbox. They record as a selector now,
+  as in a chain.
+- **An Outline row's value inside a regex matches the row replaying.**
+  Playwright writes exact text as a regex (`filter({ hasText: /^Garden
+  Queen$/ })`); that value becomes the row's `<param>` as well, and replay
+  writes the next row's value escaped, quotes, slashes and brackets
+  included.
+- **A secret inside a selector or an assertion's pattern is masked where
+  it is written escaped** (a quote in a quoted name, a `.` in a regex), and
+  replay writes it back the same way. A short one next to an escape
+  (`\buat\b`) counts as standing alone.
+- **The recorder reads only the code Playwright ran.** A snapshot or a
+  response body that showed Playwright code (a docs page, a code viewer)
+  was recorded as actions.
+- **Only tools a recording can hold reach the agent**, whatever
+  `PLAYWRIGHT_MCP_CAPS` turns on: an allowlist of the tools that record or
+  only read replaces the list of hidden ones, so coordinate mouse,
+  storage, cookie and route tools stay out.
+- **A heal checks a network assertion against the response that came
+  late.** The failed replay's response log keeps listening through the
+  heal, so a response that arrived just after replay gave up counts.
+- **A cut browser result never splits an emoji**, which made Codex drop
+  the whole result.
+- **`saffron login antigravity` writes through a settings link** (a
+  dotfiles repo, even before the file exists there, through chains and
+  linked folders) and keeps the file's permissions; a link into a
+  read-only place (a Nix store) is left as it is, with the two rules to
+  add named.
+- **The editor keeps each nested project's vocabulary**, built once, in
+  editors that watch its files for it (VS Code, JetBrains), and a changed
+  file refreshes every open project's diagnostics. A project whose build
+  fails keeps its last good one, and coming back to a project checks its
+  documents again. Opening a file it already read (go to definition)
+  rebuilds nothing.
+- **A browser action a Codex, Antigravity or Cursor agent gave up on is
+  still recorded.** Playwright finishes a click or a goto whatever the
+  agent does meanwhile, so the recording keeps what the browser did.
+
+Editors (VS Code extension 0.3.3, JetBrains plugin 0.3.3)
+
+- **A project nested inside the folder the IDE opened.** Both editors
+  now find the nearest `saffron.config.json` above a file and treat that
+  folder as the project: runs start there, status, proposals, `.env` and
+  the step vocabulary come from there, and the project's own `saffron`
+  install is used. JetBrains shows a chooser in the tool window title
+  when the IDE root holds several projects, and adds a Working directory
+  to run configurations. Before, a run from a nested project wrote
+  `.saffron/` into the parent folder with default settings and recorded
+  everything again.
+- **JetBrains: nothing runs in a project you have not trusted.** No
+  status, runs, diffs or language server until the project is trusted;
+  the tool window says so and loads as soon as you trust it. The
+  language server and the Run menu entries appear only inside a Saffron
+  project (a `saffron.config.json`, an installed `saffron-ai`, or
+  `.saffron` files), so a Cucumber or Behat folder of `.feature` files
+  no longer launches `npx -y`. A nested project is left out of the outer
+  project's file list and never run from the outer root.
+- **VS Code on Windows: runs no longer pass through a `.cmd` shim**,
+  which cmd.exe re-parses. The installed CLI runs with node directly;
+  the npx fallback escapes every argument.
+- **JetBrains: reused run configurations no longer keep a Re-record tick
+  from an earlier use.** `.feature` files appear in the Files tab and
+  run from the context menu. A new `login` command takes the provider.
+  Links in the Dashboard open in your browser. The tool window refreshes
+  only for Saffron runs and changes inside the project.
+- **Both editors list proposal files the runner cannot read**, with the
+  reason, and count them as pending. A run with no reported cost shows
+  as not reported, never as $0.
+- **VS Code:** the panel keeps the project you were on when a webview,
+  diff or image takes focus, and Select Project lists nested projects;
+  Run Tagged keeps tags written with parentheses such as `@issue(42)`;
+  stale status after a run is gone; status errors show the
+  error line, not the last stack frame; lines inside docstrings are not
+  read as steps; `Example:` is a scenario keyword; Re-record works with
+  Replay only ticked; `{data:...}` flags an empty list and a header-only
+  CSV; Run Tagged skips tags `--filter` cannot express and says which;
+  diagnostics clear when a file closes or is deleted.
+
 ## 0.9.1
 
 Released 2026-09-29.

package/README.md

@@ -4,7 +4,7 @@
 
 Free to use · [npm: saffron-ai](https://www.npmjs.com/package/saffron-ai) · [site](https://saffron-ai.io)
 
-Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude Agent SDK + Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**. A human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
+Write tests in plain language, as `.saffron` files or standard `.feature` files. `.saffron` is Saffron's own test language and goes further than Gherkin: step sets for reusable flows, `{env:VAR}` tokens that keep secrets out of recordings and reports, and `{data:...}` references to data files. There are no step definitions and no glue code. On the first run, an AI agent (Claude by default, or Codex, Google AI or Cursor on the subscription you already have, driving Playwright MCP) executes each scenario in a real browser and records what it did into a **cache**. A human-reviewable JSON action list committed to git. Every run after that, Saffron replays the cache in a real browser: **zero AI calls, zero tokens**, at the speed of the browser and your app. The cache is Saffron's own JSON, not a generated script; replay runs on Playwright's browser engine, and Playwright MCP is only how the agent operates the browser while it records or heals.
 
 When the UI changes and a cached step fails at runtime, the agent takes over mid-execution with full scenario context, adapts, and finishes the run, then files a **cache proposal** you accept or reject, like a snapshot test update. Reports state exactly what was adapted and what it cost.
 
@@ -16,6 +16,7 @@ When the UI changes and a cached step fails at runtime, the agent takes over mid
 - **Caches are git artifacts.** Proposals show diffs, including a suggested feature-file edit when the written steps no longer match reality, and nothing is committed without you.
 - **Verified proposals.** Every recording/heal is proof-replayed zero-AI before it's filed (with one bounded refinement pass on failure), so proposals arrive stamped `verified ✓` or honestly `UNVERIFIED ✗`.
 - **Honest cost reporting.** Reports break out prompt-cache reads/writes (the real bill of agent sessions) next to the in+out token count, per scenario and in totals, and name the models used. On an API key the dollars are what was billed; on a Claude subscription the headline is the 5-hour plan window before and after the run, with the dollars as the API-equivalent.
+- **Your subscription, your choice of agent.** Recording and healing run on the plan already logged in on the machine: Claude (default), Codex on a ChatGPT plan, Google AI Pro/Ultra through the Antigravity CLI (experimental), or Cursor. Set `"provider"` in the config; each one runs through its vendor's official client, and every provider records through the same recorder, so the caches are identical in kind.
 - **Trend memory.** Every run appends to `.saffron/history.jsonl`; reports show deltas vs the previous run and 20-run sparklines, and Saffron flags **chronic scenarios** (healing repeatedly. Re-record instead of paying again) plus recurring failure themes.
 
 ## Step reuse: new scenarios get cheaper as your suite grows
@@ -62,7 +63,7 @@ node examples/demo-app/server.mjs 4173 &
 node dist/cli/index.js -p examples run
 
 # delete a cache and watch the agent re-record it (needs Claude Code auth
-# or ANTHROPIC_API_KEY)
+# or ANTHROPIC_API_KEY, or another provider's login: see "provider" below)
 rm examples/.saffron/cache/login/successful-login.json
 node dist/cli/index.js -p examples run
 node dist/cli/index.js -p examples accept --all
@@ -95,6 +96,7 @@ Configuration lives in `saffron.config.json` at your project root:
   "pollIntervalMs": 100,
   "retries": 1,
   "screenshot": "viewport",
+  "provider": "claude",
   "model": "claude-sonnet-5",
   "maxTurns": 100,
   "storageState": ".auth/state.json",
@@ -109,6 +111,15 @@ Configuration lives in `saffron.config.json` at your project root:
 }
 ```
 
+`provider` (also `--provider`) picks whose agent records and heals:
+`claude` (default), `codex` (ChatGPT plan; `npm i -D @openai/codex-sdk`),
+`antigravity` (Google AI Pro/Ultra through Google's `agy` CLI,
+experimental; `npx saffron login antigravity` once) or `cursor`
+(`npm i -D @cursor/sdk`, then `npx saffron login cursor`). `model` takes
+that provider's model names. `npx saffron login` says what a provider
+still needs on this machine. Every sign-in flow is in
+[docs/PROVIDERS.md](docs/PROVIDERS.md).
+
 `storageState` (also `--storage-state`) points to a Playwright storage-state
 JSON (cookies + localStorage) applied to every scenario's browser context, so
 cached replays *and* the agent start authenticated. Generate one with