saffron-ai 0.3.0 → 0.4.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "saffron-ai",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "Gherkin-native AI-fallback test runner: zero-token cached replay, runtime AI healing, honest reports",
   "scripts": {
     "test": "vitest run",
@@ -54,6 +54,7 @@
   },
   "files": [
     "dist-pkg",
+    "skills",
     "README.md",
     "LICENSE"
   ]

package/skills/saffron/SKILL.md

@@ -0,0 +1,152 @@
+---
+name: saffron
+description: Write, run, and maintain end-to-end UI tests with Saffron — a Gherkin-native runner where an AI agent records each scenario once and every later run replays at zero tokens. Use when creating or editing .feature / .saffron files, defining StepSets, reviewing .saffron/cache proposals, running `saffron run`, or when the user mentions Saffron, Gherkin scenarios, step sets, cached replay, or `saffron.config.json`.
+license: Saffron Free Use License v1.0 (see the saffron-ai package LICENSE)
+metadata:
+  author: saffron-ai
+  homepage: https://saffron-ai.lovable.app
+---
+
+# Saffron
+
+Saffron runs plain-Gherkin scenarios in a real browser. There are **no
+step definitions and no glue code**: on the first run an AI agent performs
+each step and records what it did into a JSON cache; every later run
+replays that cache with plain Playwright — zero AI calls, zero tokens,
+~100 ms per scenario. When the UI drifts, the agent heals the failing
+step mid-run and files a reviewable proposal.
+
+Your job when writing tests is to make that economy work. Three rules
+matter more than everything else:
+
+## 1. Exact step text is the cache identity — reuse the vocabulary
+
+A step is looked up by its exact wording. `When I visit the login page`
+and `When I open the login page` are two different steps: the second one
+costs a fresh AI recording (~$0.10–$2) even though the first already
+replays for free.
+
+**Before writing any step, look at what the project already knows:**
+
+```bash
+npx saffron steps                 # every step: ● recorded, ● divergent, ○ unrecorded
+npx saffron steps "login"         # search
+npx saffron steps --json          # machine-readable
+```
+
+Prefer a `●` recorded wording verbatim. Quoted values may differ freely
+(`I enter "admin"` seeds from `I enter "bob"`), so parameterize with
+quotes rather than inventing new phrasings. `saffron init` registers the
+`saffron` MCP server with Claude Code, Cursor and VS Code — its
+`search_steps` / `list_step_sets` tools give the same answer without a
+shell.
+
+## 2. `Then` steps are sacred — write them as the verdict
+
+Saffron never heals an assertion: the agent may help *reach* a `Then`,
+never make it pass. So `Then` lines must state exactly what must be true,
+with stable text — never volatile values (prices, dates, counters). Put
+the final assertions **last**; the trailing block of `Then` steps is
+strict under every policy. Use `Given`/`When` for actions.
+
+## 3. Never bake volatile or secret values into files
+
+- Secrets: `{env:VAR}` — `When I enter "{env:ADMIN_PASSWORD}" in the password field`.
+- Dates: write intent, not literals — "1 day from today" records as `{date+1}`.
+- Dynamic display values: capture and compare — `I record the total as "first"` … `"first" should differ from the displayed total`.
+
+## Writing a scenario — the shape that records cleanly
+
+```gherkin
+Feature: Checkout
+
+  Background:
+    Given I open the application
+    And I accept cookies if prompted          # conditional steps are fine
+
+  @smoke
+  Scenario: Guest can complete checkout
+    When I add "Blue Widget" to the cart
+    And I place the order and wait for the order API to return 201
+    Then I should see "Order placed"
+    And the order request should have returned 201
+```
+
+Guidelines the recorder rewards:
+
+- One intent per step; 5–10 steps per scenario. Long scenarios heal badly.
+- Name the page or control in the step (`on the checkout page`, `the "Save" button`).
+- Data belongs in tables and doc strings, not in prose (see references).
+- Conditional wording (`if prompted`) records as a no-op when absent.
+- Real-world furniture just works — say it: dialogs ("and confirm the
+  dialog"), uploads ("upload the file "x.txt" as the attachment"),
+  drag-and-drop ("drag the card onto the done column"), iframes
+  (interact normally), network waits ("wait for the order API to return
+  201", "wait until the job status API reports "READY"").
+
+## Repeated step groups → StepSets (`.saffron` files only)
+
+`.saffron` is a superset of `.feature`: rename the file, nothing else
+changes, and you gain the `StepSet:` keyword.
+
+```gherkin
+StepSet: Complete guest information            # define — colon, like Scenario:
+    Given I am on the guest information page
+    When I enter guest name "Chathuranga"
+    And I click the continue button
+    Then I am not on the guest information page
+
+Scenario: Checkout happy path
+    Given I am on the cart page
+    StepSet Complete guest information         # invoke — no colon, like a step
+    Then I am on the checkout page
+```
+
+- Definition: `StepSet: <name>`. Invocation: `StepSet <name>` — the bare
+  name without the keyword is a silent parse error.
+- Names are project-wide unique; sets resolve across files; no nesting.
+- Application-wide flows (login, cookie banner) live in a sets-only
+  library file: `features/shared.steps.saffron` (still needs a
+  `Feature:` header; it yields no runnable scenarios).
+- Start a set with a guard step, end it with an exit assertion.
+
+## The workflow
+
+```bash
+npx saffron run                          # record misses, replay hits, heal failures
+npx saffron run --filter @smoke          # by tag
+npx saffron run --no-agent               # replay only (CI without AI access)
+npx saffron accept                       # list pending proposals
+npx saffron accept --all                 # promote verified proposals to caches
+npx saffron report                       # open the HTML report
+```
+
+Results: **green** = cached pass · **yellow** = passed with AI recording
+or adaptation, a proposal awaits review · **red** = failed. Proposals
+arrive stamped `verified ✓` (zero-AI proof replay passed) or
+`UNVERIFIED ✗`; accept the former, investigate the latter. Commit
+`.saffron/cache/` like snapshots; git-ignore `.saffron/reports/`.
+
+**Never hand-edit files under `.saffron/cache/` or `.saffron/proposals/`**
+unless deliberately authoring a manual cache (`"recordedBy": "manual"`).
+To change behavior, change the `.feature`/`.saffron` text and let the run
+re-record (mostly seeded from existing step recordings).
+
+## Anti-patterns that cost money or hide bugs
+
+| Don't | Do |
+|---|---|
+| Invent a synonym for an existing step | Reuse the `●` wording from `saffron steps` |
+| Assert inside a `When` ("When I see the dashboard") | Act in `When`, assert in `Then` |
+| `Then the price is "NOK 1,148"` | `Then the price should match "NOK [\d,]+"` intent, or capture + compare |
+| Put a password literal in a step | `{env:VAR}` |
+| Sleep/wait "for 3 seconds" | Wait for a UI signal or an API response |
+| One 30-step scenario | Several 5–10 step scenarios + StepSets |
+| Edit cache JSON to fix a test | Edit the scenario text; re-run |
+
+## References
+
+- [Syntax cheat sheet](references/syntax.md) — tables, doc strings,
+  outlines, secrets, dates, network waits, page furniture, StepSets.
+- [Configuration & CI](references/config.md) — `saffron.config.json`,
+  flags, `--strict`, cross-browser, workers, heal model, secrets setup.

package/skills/saffron/references/config.md

@@ -0,0 +1,90 @@
+# Configuration, CLI and CI
+
+## Project layout
+
+```
+your-project/
+  features/               .feature and .saffron files
+  saffron.config.json
+  .saffron/
+    cache/                committed replay caches      → commit
+    proposals/            pending AI proposals         → review, then gone
+    history.jsonl         one line per run (trends)    → commit recommended
+    reports/              latest.html / latest.json    → git-ignore
+```
+
+## `saffron.config.json`
+
+```json
+{
+  "baseURL": "https://stage.your-app.com",
+  "features": "features",
+  "actionTimeoutMs": 5000,
+  "retries": 1,
+  "model": "claude-sonnet-5",
+  "healModel": "claude-haiku-4-5",
+  "maxTurns": 100,
+  "storageState": ".auth/state.json",
+  "strict": false,
+  "assertionPolicy": "strict",
+  "verifyProposals": true,
+  "reuseSteps": true,
+  "snapshotMode": "none",
+  "browser": "chromium",
+  "workers": 1
+}
+```
+
+| Key | Meaning |
+|---|---|
+| `baseURL` | App under test; steps say "the login page", not full URLs |
+| `storageState` | Playwright storage-state JSON so replays and the agent start authenticated (`npx playwright open --save-storage=.auth/state.json <url>`) |
+| `strict` | Yellow (passed-with-adaptation) exits 1 until reviewed — cached-green-only CI |
+| `assertionPolicy` | `strict` (default) or `adaptable-mid`; the final assertion block is always strict |
+| `verifyProposals` | Proof-replay every recording zero-AI before filing (default true) |
+| `reuseSteps` | Seed new recordings from existing step recordings (default true) |
+| `snapshotMode` | `none` (default, ~60% fewer AI calls) or `full` for highly dynamic pages |
+| `browser` | `chromium` / `firefox` / `webkit` — replay runs anywhere; recording and healing use Chromium |
+| `workers` | Parallel replay workers; agent work stays sequential |
+| `healModel` | Cheaper model for heal sessions only |
+
+## CLI
+
+| Command | Purpose |
+|---|---|
+| `saffron run [paths] [--filter @tag] [--no-agent] [--strict] [--browser b] [--workers n] [--heal-model m] [--headed]` | Run; cached replay, agent on misses/failures |
+| `saffron accept [file \| --all] [--with-feature-edit] [--propagate]` | Promote proposals; `--with-feature-edit` rewrites adapted steps in the feature file; `--propagate` applies a heal's locator fix to every cache using that locator |
+| `saffron reject [file \| --all]` | Discard proposals |
+| `saffron steps [search] [--json] [--snippets]` | The step vocabulary with recorded/divergent/unrecorded badges |
+| `saffron author <prose-file>` | Draft a `.saffron` file from plain-paragraph requirements using the project vocabulary |
+| `saffron mcp` / `saffron lsp` | Vocabulary for AI assistants / language server for editors |
+| `saffron init` | Install this skill into the project's agent directories and scaffold config |
+| `saffron report` | Open the latest HTML report |
+
+Exit codes: 0 green/yellow, 1 red (or yellow with `--strict`), 2 usage /
+preflight (e.g. a missing `{env:VAR}`).
+
+## AI access
+
+Recording and healing need Claude credentials: `ANTHROPIC_API_KEY`, or a
+Claude Code login on the machine. Replay-only runs (`--no-agent`) need
+none — that is the normal CI mode once caches are committed.
+
+## Recommended CI
+
+```bash
+npx playwright install chromium
+npx saffron run --no-agent --strict     # replay committed caches; no AI, no surprises
+```
+
+Record and heal on developer machines (or a dedicated job with
+credentials), review proposals in PRs like snapshot updates.
+
+## Reading a report
+
+Per scenario: status, duration, AI calls and tokens (including prompt
+cache reads/writes — the real bill), adaptation narrative, cache diff and
+suggested feature edit for yellows, drift chips when page fingerprints no
+longer match. Trends vs. the previous run and 20-run sparklines come from
+`.saffron/history.jsonl`; **chronic** scenarios (healing repeatedly) are
+flagged — re-record those instead of paying for heals again.

package/skills/saffron/references/syntax.md

@@ -0,0 +1,167 @@
+# Saffron syntax cheat sheet
+
+Everything valid Gherkin is valid Saffron. `.saffron` files add `StepSet:`.
+
+## Steps
+
+```gherkin
+Given I am on the login page                     # action (healable)
+When I enter "standard_user" in the username field
+And I click the "Log in" button
+Then I should see the products page              # assertion (never healed)
+But I should not see an error banner
+```
+
+`And`/`But` inherit action-vs-assertion from the preceding step. The
+trailing block of assertions is always strict; under the opt-in
+`adaptable-mid` policy only mid-scenario checkpoints may be adapted.
+
+## Scenario Outlines
+
+One cache serves every row — `<param>` placeholders stay in the cache and
+resolve per row at replay.
+
+```gherkin
+Scenario Outline: Failed login shows an error
+  When I enter "<username>" in the username field
+  And I enter "<password>" in the password field
+  And I click the "Log in" button
+  Then I should see the error message "<message>"
+
+  Examples:
+    | username | password | message                    |
+    | locked   | secret   | This user has been locked. |
+    | standard | wrong    | Wrong username or password |
+```
+
+## Data tables
+
+**2-column = key/value.** Values record as `<table:key>` references:
+edit the values and replay stays free; change a key and the step
+honestly re-records.
+
+```gherkin
+When I enter the following credentials
+  | username | premium_user |
+  | password | secret2      |
+```
+
+**Wider = records** (header row + one row per record). Cells record as
+`<table:1:firstName>`, `<table:2:email>` (1-based rows). Editing any cell
+replays free; adding/removing rows or renaming headers re-records.
+
+```gherkin
+When I add the following guests
+  | firstName | email          | city   |
+  | Alice     | alice@test.com | Oslo   |
+  | Bob       | bob@test.com   | Bergen |
+```
+
+## Doc strings
+
+Content records as `<docstring>`; rewording replays free. A later
+assertion on that content follows the edit too ("the saved note should
+be shown").
+
+```gherkin
+When I leave a note
+  """
+  Please deliver after 5pm.
+  """
+Then the saved note should be shown
+```
+
+## Secrets — `{env:VAR}`
+
+```gherkin
+When I enter "{env:ADMIN_USER}" in the username field
+And I enter "{env:ADMIN_PASSWORD}" in the password field
+```
+
+Resolved from the environment (or a git-ignored `.env`; real env wins) at
+replay. Caches, proposals, reports and history contain only the token;
+a missing variable fails fast by name. Honest note: during the *first*
+recording the agent types the real value once — use rotatable staging
+credentials.
+
+## Dates and dynamic values
+
+- Say the intent: "select a check-in date 1 day from today" → recorded
+  as `{date+1}`; stays valid every day. Formats: `{date}`, `{date-3}`,
+  `{date+1:DD.MM.YYYY}`.
+- Capture and compare displayed values instead of literals:
+
+```gherkin
+When I record the displayed total as "before"
+And I add another night
+Then "before" should differ from the displayed total
+And the total should match "NOK [\d,]+"
+And the "Book" link should point to "/booking"
+```
+
+## Network-aware steps (no keywords — plain prose)
+
+```gherkin
+When I place the order and wait for the order API to return 201
+And I wait until the job status API reports "READY"
+Then the order request should have returned 201        # assertion, never healed
+```
+
+Recorded as URL-pattern + method + status (+ body pattern) matchers;
+waits are satisfied by responses from the triggering step onward, so
+"click and wait" never races. Prefer these over any fixed sleep.
+
+## Page furniture
+
+```gherkin
+When I click "Clear workspace" and confirm the dialog       # alert/confirm/prompt
+When I upload the file "examples/sample.txt" as the attachment
+When I drag the task card onto the done column
+When I enter "Great tool" in the feedback comment and send it   # inside an iframe: just interact
+Then the feedback widget should show "Thanks for: Great tool"
+```
+
+Upload paths resolve from the working directory. Links that open new
+tabs are followed automatically.
+
+## StepSets (`.saffron` only)
+
+```gherkin
+Feature: Checkout
+
+StepSet: Complete guest information
+    Given I am on the guest information page
+    When I enter guest name "Chathuranga"
+    And I click the continue button
+    Then I am not on the guest information page
+
+Scenario: Checkout happy path
+    Given I am on the cart page
+    StepSet Complete guest information
+    Then I am on the checkout page
+```
+
+| Form | Syntax | Notes |
+|---|---|---|
+| Define | `StepSet: <name>` | Colon, like `Scenario:`; steps indented below |
+| Invoke | `StepSet <name>` | No colon; sits anywhere among steps |
+
+- Expanded at parse time: inlined steps cache and seed like ordinary
+  steps; editing a set makes every caller honestly stale (re-recorded
+  mostly seeded); heal edits route to the set definition.
+- Names project-wide unique (also catches the colon typo at an
+  invocation). Sets may contain tables, doc strings and `<param>`
+  placeholders; no nesting; a set never runs standalone.
+- Shared flows go in `features/shared.steps.saffron` — a sets-only
+  library file with a `Feature:` header and no scenarios.
+- Assertions inside a set are checkpoints mid-scenario and part of the
+  strict final block when the set is invoked last.
+
+## Tags
+
+```gherkin
+@smoke @checkout
+Scenario: ...
+```
+
+`saffron run --filter @smoke`.