saffron-ai 0.6.0 → 0.7.0

package/CHANGELOG.md

@@ -4,6 +4,199 @@ All notable changes to Saffron, in one place: the `saffron-ai` runner (npm),
 the VS Code extension (`ChathurangaJayasinghe.saffron-vscode`) and the
 JetBrains plugin (`ai.saffron.jetbrains`). Dates are publish dates.
 
+## 0.7.0
+
+Released 2026-09-19.
+
+New: `saffron diff`, `saffron prune`, a configurable `pollIntervalMs`, and a
+configuration reference in the docs. The rest of this release is correctness
+work from several rounds of external review.
+
+**Upgrading.** Three fixes can change what an existing suite does, all in
+the direction of telling the truth:
+
+- A scenario that was green because replay quietly clicked a near-match
+  ("Delete all orders" for a missing "Delete"), or because a `Then` step
+  never checked anything, now fails and heals or asks to be re-recorded.
+- Feature files in subdirectories get a new cache directory name (a short
+  digest of their path is appended), so their scenarios record again once.
+  Files directly under the features root are unaffected. `saffron prune`
+  lists the old directories afterwards.
+- `saffron status --json` gains `cacheState` and `orphans`; nothing was
+  removed.
+
+Runner
+
+Correctness fixes from an external review of the runner. Four of these
+could let a run look green or yellow when the goal was never checked.
+
+- **Secrets could reach your feature files.** Machine-applicable feature
+  edits were the one agent output that skipped `{env:VAR}` masking, so
+  `saffron accept --with-feature-edit` could write a resolved password into
+  a scenario. Feature edits, the narrative and the suggested edit are now
+  masked everywhere they are stored or reported.
+- **An assertion that recorded nothing now fails.** A `Then` step with an
+  empty action list used to replay as a pass, so a scenario could be green
+  without its verdict ever being checked. Replay fails such a step, and the
+  recording is filed unverified. *This can turn an existing green scenario
+  red: that assertion was never being checked. Re-record it.*
+- **Negative assertions consider every recorded locator.** `expectNotVisible`
+  returned as soon as one candidate was hidden, so a real failure could hide
+  behind a missing primary locator while a fallback was visible.
+- **Recorded names match exactly first.** A step recorded against "Save"
+  could click "Save draft" when it came first in the DOM. Substring matching
+  remains the fallback for reading the page. The agent is no longer told to
+  shorten names.
+- **Cache files are unique per scenario.** The cache directory now includes
+  the feature file's directories, so two same-named files in different
+  folders stop overwriting each other's recordings. Flat layouts are
+  unchanged. A run refuses to start if two scenarios would still share a file.
+- **Suspicious recordings are saved unverified.** The check ran after the
+  proposal was written, so the file said verified while the run said
+  otherwise, and `accept --all` took it.
+- **An unverified proposal shows the proof replay's timeline**, not the
+  agent session's, so the report says which step could not replay instead
+  of marking every step passed.
+- **A step that changes between an action and an assertion is stale.** Cache
+  matching compared the step's words but not its keyword, so turning
+  `When I see the total` into `Then I see the total` kept the old recording
+  and replayed the action instead of checking anything.
+- **Waiting for a new tab happens where a tab opens.** Every click paid a
+  150ms wait, which taxed scenarios that never use tabs and still lost the
+  race with a slower popup. The proof replay now records which action opens
+  a tab, and only that action waits, for as long as any other action may
+  take. A five-step login scenario replays in about a third of the time.
+- **`saffron status` distinguishes a valid cache from a stale one.** It
+  reported `cached` by asking whether a file existed, so a scenario whose
+  text had changed still looked recorded. Each scenario now carries
+  `cacheState`: `valid`, `stale`, `invalid` or `missing`, and the printed
+  view marks stale ones.
+- **`saffron lsp` no longer reports a valid step set as defined twice** in
+  projects whose path contains a space: it compared a file system path with
+  a percent-encoded URL path.
+
+- **`saffron diff` shows what a proposal changes**: the committed recording
+  against the proposed one, action by action, with a rewritten action shown
+  as one before-and-after pair and changed assertions counted separately.
+  The narrative is the agent's account of what it did; this is what a
+  reviewer is approving. It also appears in the HTML report and in both IDE
+  panels.
+- **An assertion step must contain an assertion.** The earlier guard only
+  rejected an empty action list, so a `Then` that recorded a `waitFor` and
+  nothing else still passed.
+- **A control that is found but cannot be used no longer hands the action to
+  a different control.** With a disabled "Save" next to an enabled "Save
+  draft", replay clicked the draft button. An interaction now stops at the
+  candidate that matched, so drift goes to the failure and healing path
+  instead of quietly doing something else.
+- **Negative assertions check every match of every locator.** Checking only
+  the first match let a hidden first element vouch for a visible second one.
+  A locator that errors no longer counts as proof of absence.
+- **Secrets are discovered through Examples rows.** A `{env:VAR}` reaching a
+  table cell or doc string through a placeholder was invisible to masking,
+  so it could survive into recordings, reports and feature edits.
+- **Cache identity follows the path, not its spelling.** `admin/login.saffron`
+  and `admin-login.saffron` produced the same cache directory; nested files
+  now carry a digest of their path. The collision check also looks at the
+  whole project, because two features run separately never met before.
+- **A recording that says a tab opens must see one.** The marker was
+  advisory: when no tab appeared the timeout was swallowed and the next
+  assertion ran against the old page. Tab openers are also captured while
+  recording now, not only during the proof replay, which could miss them.
+- **An interaction never substitutes a different control.** A recorded name
+  that no longer matches exactly could still act on any control containing
+  it, so a missing "Delete" could click "Delete all orders" and the run
+  stayed green. Interactions now require the exact name; assertions and
+  captures keep the substring fallback, because reading the page is not
+  acting on it. *A scenario whose control was renamed now fails and heals
+  instead of passing.*
+- **The proposal diff compares actions, not their descriptions.** Two
+  actions that read the same could differ where it counts: dropping a
+  response body check or repointing a selector from `#save` to `#delete`
+  both reported no change. Actions are compared field by field, and a
+  rewritten action now lists exactly which fields moved.
+- **Repeated steps stay separate in the diff.** Steps were matched by their
+  wording, so a scenario that adds the same item twice showed no change when
+  the proposal dropped one of them.
+- **A popup that arrives late is pinned on the action that opened it.** The
+  opener marker waited for the next recorded action, so a slow window was
+  blamed on whatever the agent did afterwards, and replay then waited for a
+  tab that action never opens. Only an interaction can be credited: a wait
+  or an assertion recorded between the click and the popup no longer takes
+  it, and the marker now follows the agent's actual tool calls instead of
+  inferring them. Any action marked as opening a tab waits for one, not
+  only clicks. A popup nobody can be credited with is dropped rather than
+  passed to a later action: one that arrives while the agent is idle and
+  the last interaction is too old, one that lands during a call that failed
+  or could have opened it itself, and one the agent opened with a new tab.
+  Only known observation tools (a snapshot, a wait, listing tabs) may vouch
+  for the click before them, and failed tool calls are now drained so they
+  cannot hold an attribution open.
+- **`saffron prune` removes recordings nothing owns.** Deleting a scenario
+  or a feature file left its cache behind, where it stayed in git and no
+  reviewer could tell it from a live one. `saffron prune` lists every
+  orphaned cache and proposal with the reason, deletes nothing without
+  `--yes`, and `--check` exits 1 for CI. Ownership follows the path, so a
+  stale or unreadable cache that a live scenario claims is kept, and while
+  any feature file fails to parse prune refuses to judge anything.
+  `saffron status --json` carries the same list for IDE panels.
+- **`pollIntervalMs` is configurable** (default 100ms, minimum 10). It sets
+  how often a polled assertion re-checks the page inside `actionTimeoutMs`:
+  lower reacts sooner, higher costs less on a busy CI box.
+
+Docs
+
+- **A configuration reference**, grouped by what you are tuning (timing,
+  project, runtime, the agent, gates) with every option's default, replacing
+  the paragraph that listed the keys in prose. It ends with how timing
+  actually works: Playwright's actionability wait, then polled assertions,
+  then action retries, and why there is no whole-scenario retry.
+
+Editors (JetBrains plugin 0.2.2, VS Code extension 0.2.4)
+
+- JetBrains plugin: **Accept Selected with nothing ticked accepted every
+  proposal.** Blank paths mean `--all` to the runner, so the deliberate
+  review workflow could be bypassed by clicking the wrong button. It now
+  does nothing and says so; bulk acceptance stays its own action.
+- Both IDEs list **recordings no scenario owns any more**: a JetBrains
+  *Orphans* tab (grouped by cache and proposal, double-click to open one,
+  *Remove All* after a confirmation) and a VS Code *Orphaned recordings*
+  section with the same action in its title bar. Both name the reason, and
+  both say plainly when the installed runner is too old to report them
+  rather than claiming there are none. `prune` is also a JetBrains run
+  configuration command, where listing is the default.
+- JetBrains plugin: **the Proposals tab could show an old diff beside a new
+  narrative.** Diffs were cached by file name, and a new run overwrites a
+  proposal under the same name, so Accept could promote something the
+  reviewer never read. A refresh now invalidates every diff, a late result
+  from before the refresh is discarded, and the cache lives on the UI thread.
+- JetBrains plugin: **a path with a space stays one path.**
+  `features/order checkout.saffron` became two arguments, even for
+  right-clicked and ticked files. Paths are parsed like a command line, and
+  what the plugin writes itself is quoted to match.
+- VS Code extension: **Run selected ran project B's files from project A**
+  in a multi-root workspace, with A's config and install. Selections are now
+  grouped by owning folder, one run per project, from the Explorer too. The
+  status-backed sections name the project they show, and *Select project*
+  switches it, instead of silently using the first folder. Switching
+  clears those sections at once rather than when the new status arrives:
+  rows left on screen stayed clickable, so a proposal ticked in project A
+  could be accepted in project B when both had the same relative file.
+  Ticks are bound to the project that listed them, and accept, reject,
+  run tagged and prune refuse to act while a status is still loading.
+- VS Code extension: **the proposal diff could not start on Windows**, where
+  Node refuses to launch a `.cmd` without a shell. Status and diff now share
+  one launcher: the installed CLI runs under node with a plain argument
+  array, and only the npx fallback goes through cmd.exe, fully escaped.
+- Both IDEs show the **proposal diff**: JetBrains under the narrative in the
+  Proposals tab, VS Code by opening the diff as a document when a proposal is
+  clicked or from its inline action.
+- VS Code extension: runs the CLI as a **task with an argument array and the
+  owning folder's working directory**, instead of writing a command line
+  into a shared terminal. A file name with a dollar sign or a backtick is no
+  longer shell syntax, and a file from a second workspace folder runs
+  against that folder's config and install.
+
 ## 0.6.0
 
 Released 2026-09-18.
@@ -20,6 +213,12 @@ Runner
 - When a pending proposal fails its zero-AI replay and the agent records
   again, the run now says which step failed and why.
 
+Editors
+
+- VS Code extension 0.2.3: the Marketplace listing is named "Saffron",
+  its short description fits without being cut off, and the overview
+  introduces the tool and how to get started before listing features.
+
 ## 0.5.7
 
 Released 2026-09-17.

package/README.md

@@ -90,6 +90,7 @@ Configuration lives in `saffron.config.json` at your project root:
   "baseURL": "http://localhost:4173",
   "features": "features",
   "actionTimeoutMs": 5000,
+  "pollIntervalMs": 100,
   "retries": 1,
   "model": "claude-sonnet-5",
   "maxTurns": 100,