saffron-ai 1.3.0 → 1.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +788 -0
- package/dist-pkg/cli.js +396 -376
- package/package.json +1 -1
- package/skills/saffron/SKILL.md +13 -2
- package/skills/saffron/references/config.md +3 -2
- package/skills/saffron/references/syntax.md +9 -0
- package/templates/playwright/saffron-helpers.ts +399 -74
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "saffron-ai",
|
|
3
|
-
"version": "1.3.
|
|
3
|
+
"version": "1.3.2",
|
|
4
4
|
"description": "End-to-end tests written in plain language: an AI agent records each scenario once, replays run at zero tokens, and broken replays heal into proposals you review",
|
|
5
5
|
"scripts": {
|
|
6
6
|
"test": "vitest run",
|
package/skills/saffron/SKILL.md
CHANGED
|
@@ -3,14 +3,14 @@ name: saffron
|
|
|
3
3
|
description: Create, translate, or edit Saffron test scenarios in .saffron or .feature files, browser UI tests and HTTP API tests (@api) alike, including validating their recordings and replay. Use when asked to author or fix these test files in a Saffron project. Do not activate for ordinary application development, generic Cucumber projects, or requests only to run existing tests, inspect reports, review proposals, or configure Saffron.
|
|
4
4
|
license: Saffron Free Use License v1.0 (see the saffron-ai package LICENSE)
|
|
5
5
|
metadata:
|
|
6
|
-
saffron-ai: "1.3.
|
|
6
|
+
saffron-ai: "1.3.2"
|
|
7
7
|
author: saffron-ai
|
|
8
8
|
homepage: https://saffron-ai.io
|
|
9
9
|
---
|
|
10
10
|
|
|
11
11
|
# Saffron
|
|
12
12
|
|
|
13
|
-
This skill describes saffron-ai 1.3.
|
|
13
|
+
This skill describes saffron-ai 1.3.2 (see the compatibility page: https://saffron-ai.io/docs/compatibility).
|
|
14
14
|
|
|
15
15
|
Apply this workflow only when asked to create, translate, or edit Saffron
|
|
16
16
|
scenarios in `.saffron` or `.feature` files. A Saffron mention or an
|
|
@@ -366,6 +366,17 @@ or weakening assertions. For CI artifacts, browser matrices, provider
|
|
|
366
366
|
setup, and other configuration needed by the requested test, read
|
|
367
367
|
[Configuration & CI](references/config.md).
|
|
368
368
|
|
|
369
|
+
## When Saffron cannot express a value: a small supporting script
|
|
370
|
+
|
|
371
|
+
Use Saffron itself first: `{date...}` tokens (ranges and locales included), captures and comparisons, `{data:...}` and `{unique:...}`, the checks in the syntax reference, and `setup`/`teardown`. A recording that freezes a value or checks a pattern any value would pass is a recording to reword or re-record, not a missing capability. Reach for a script only when the runner cannot express a requirement at replay, and ask the user first when it would add a dependency, a hook, or a file outside the project's existing scripts folder.
|
|
372
|
+
|
|
373
|
+
- Keep it small: one file in the project's existing scripts folder, in its runtime and module style, using the standard library or dependencies already installed. Clicks and checks stay in the scenario; the script only prepares data.
|
|
374
|
+
- Expected values come from the test's inputs or an independent source (a contract, a fixture), never from the output being checked. Generate volatile values at run time, and state locale, time zone, format and day-boundary assumptions in the script.
|
|
375
|
+
- Run it as a `setup` command in `saffron.config.json`, keeping the hooks already there. It writes a git-ignored file under `data/`, which scenarios read through named `{data:...}` tokens: Saffron reads data after setup, so every replay sees fresh values. Commit the script and its configuration, never what it generates, and never write secrets into it or its output. On failure it exits non-zero with a message: setup failing stops the run before any scenario. `--no-hooks` skips it: scenarios then read its last output, or stop naming the missing file; say so.
|
|
376
|
+
- Never weaken a check to fit, and never hand-edit caches or proposals.
|
|
377
|
+
- Check it in proportion: its logic at the boundaries it handles (month end, year end, the time zone), the recording's checks and `{data:...}` tokens reviewed, only verified proposals accepted, and the scenario passing with `--no-agent`.
|
|
378
|
+
- Report which files you wrote, which the script generates and which recordings Saffron made, why the script is needed, and which Saffron capability could replace it.
|
|
379
|
+
|
|
369
380
|
## Anti-patterns that cost money or hide bugs
|
|
370
381
|
|
|
371
382
|
| Don't | Do |
|
|
@@ -59,6 +59,7 @@ your-project/
|
|
|
59
59
|
| `snapshotMode` | `none` (default, ~60% fewer AI calls) or `full` for highly dynamic pages |
|
|
60
60
|
| `setup` / `teardown` | shell command or list, run once before / after the run in the project root (seed and clean test data). Failed setup: exit 2, nothing runs. Teardown always runs. They receive `SAFFRON_BASE_URL` and `SAFFRON_ENV`. `--no-hooks` skips them; `hookTimeoutMs` (default 300000) bounds each |
|
|
61
61
|
| `trace` | `off` (default), `on` or `retain-on-failure`: keep an execution trace per scenario; `saffron trace` opens it as a replay with the page inspectable at every step |
|
|
62
|
+
| `video` | `off` (default), `on` or `retain-on-failure`: keep a video of each scenario's run, `video.webm` beside its screenshots; also `--video` |
|
|
62
63
|
| `browser` | `chromium` / `chrome` / `msedge` / `firefox` / `webkit`: replay runs anywhere; recording and healing need one of the first three. `chrome` and `msedge` must be installed on the machine. A list runs the suite in each into one report (browser matrix; Chromium-family first) |
|
|
63
64
|
| `device` / `viewport` | A Playwright device profile (`"iPhone 15"`) and/or a `"WIDTHxHEIGHT"` size for every scenario; a desktop recording replays under it or fails at the step that differs |
|
|
64
65
|
| `timezone` | IANA zone (`"Europe/Berlin"`) the browser runs in and `{date...}` tokens count days in (default: the machine's) |
|
|
@@ -70,7 +71,7 @@ your-project/
|
|
|
70
71
|
| `model` | Model in the provider's own names (unset: the provider's default). A `--provider` run on another provider ignores it |
|
|
71
72
|
| `healModel` | Cheaper model for heal sessions only |
|
|
72
73
|
| `antigravityApiKey` | `true`: Antigravity signs in with `GEMINI_API_KEY` even beside an agy login (default: a login wins) |
|
|
73
|
-
| `ci` | Where `saffron runs` and `import --run` find CI runs: `{"provider": "github" \| "azure", "organization": ..., "project": ...}`. Unset, an Azure Repos remote means Azure Pipelines (through `az`), anything else GitHub Actions (through `gh`). `organization` and `project` are for a GitHub repository built by Azure Pipelines |
|
|
74
|
+
| `ci` | Where `saffron runs` and `import --run` find CI runs: `{"provider": "github" \| "azure", "organization": ..., "project": ...}`. Unset, an Azure Repos remote means Azure Pipelines (through `az`), anything else GitHub Actions (through `gh`). `organization` and `project` are for a GitHub repository built by Azure Pipelines. `cacheRuns` and `cacheMB` cap the runs viewed into `.saffron/ci/` (default 10 and 1024) |
|
|
74
75
|
| `api` | The API under test: `{"baseURL": ..., "contract": "api/openapi.json", "transcript": "on-failure"}`. `baseURL` is where API steps' relative paths go (unset: the run's `baseURL`); `contract` is the pinned OpenAPI 3 document schema checks validate against (`saffron api pin` writes it), and a change to it, or to which file it names, makes the recordings checked against it stale; `transcript` keeps each scenario's requests and responses for the report and `saffron trace`: `"on-failure"` (default), `"on"` or `"off"` (statuses and timings only); `services` names more APIs, each `{"baseURL": ..., "contract": ...}`, which a request names instead of a host (`--api-url name=<url>` points one elsewhere for a run). See [API tests](api.md) |
|
|
75
76
|
|
|
76
77
|
## CLI
|
|
@@ -89,7 +90,7 @@ your-project/
|
|
|
89
90
|
| `saffron mcp` / `saffron lsp` | MCP tools (`search_steps`, `list_step_sets`, `project_status`, `run_plan`, `api_contract`, `verify`) for AI assistants / language server for editors |
|
|
90
91
|
| `saffron api pin <source> [--out file] [--service name]` | Pin the API contract: fetch an OpenAPI 3 document from a Swagger UI page, a document URL or a file, and write it as JSON (default `api/openapi.json`, or `api/<name>.json` with `--service`). Add it to the config as `api.contract`, or as `api.services.<name>.contract` |
|
|
91
92
|
| `saffron init [--examples] [--agents list]` | Install this skill into the project's agent directories, register MCP, add an AGENTS.md block, scaffold config; `--examples` adds the Saucedemo example suite |
|
|
92
|
-
| `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs
|
|
93
|
+
| `saffron report [--merge [paths]] \| --run <id> [--provider p] [--allow-fork] [--max-download MB] [--json \| --panel]` | Open the latest HTML report; `--merge` combines a complete set of shard reports into `latest.json`, `latest.html` and one history line. `--run` views a CI run's results without importing: downloaded once into `.saffron/ci/` (newest 10 runs and 1 GB unless `ci.cacheRuns` and `ci.cacheMB` say otherwise), checked like an import, proposals judged importable or not with the reason, nothing filed. No bundle exits 0 with `reasonKind` `running`, `expired` or `none`; over `--max-download` exits 2 with `kind: "too-large"` and `sizeBytes`. `saffron import --run` reuses the download. `--panel` prints the run's results as the one HTML page the editors' CI results panel shows |
|
|
93
94
|
| `saffron dashboard [--stdout] [--no-open]` | The suite's quality from the run history (failing and for how long, flaky, repeatedly healed, never run, slowest, trends, per tag); writes and opens `.saffron/reports/dashboard.html`, `--stdout` prints it |
|
|
94
95
|
| `saffron export [-o file] [--branch name [--base branch]] [--report-only]` | Pack the pending proposals, the run's report, screenshots and traces into one bundle (default `.saffron/reports/saffron-bundle.zip`). `--report-only` writes a summary instead (report and screenshots only, `saffron-summary.zip`): upload it as a `saffron-summary*` artifact on every run so any run's results can be viewed; a run that left only summaries has nothing to import. In CI write it with `--out <file outside .saffron>` and upload that file as an artifact (`upload-artifact` skips hidden folders). `--branch` also pushes the proposals as a new branch and opens a draft pull request (GitHub through `gh`, Azure DevOps through `az`); exit 1 only when the branch is pushed but the pull request could not be opened |
|
|
95
96
|
| `saffron runs [--branch b] [--limit n] [--provider github\|azure] [--json]` | The branch's CI runs, with how many of each run's proposals are pending here; exit 2 when `gh` or `az` is missing or signed out |
|
|
@@ -18,6 +18,15 @@ But I should not see an error banner
|
|
|
18
18
|
trailing block of assertions is always strict; under the opt-in
|
|
19
19
|
`adaptable-mid` policy only mid-scenario checkpoints may be adapted.
|
|
20
20
|
|
|
21
|
+
A check on the page's address is recorded the way the step says it: that
|
|
22
|
+
the URL contains a value (`Then the URL contains "/orders"`, the default),
|
|
23
|
+
that it no longer holds one (`Then the URL no longer holds the login
|
|
24
|
+
token`), or that its path is exactly one (`Then I am on the /account path
|
|
25
|
+
exactly`, its query and fragment aside). Write "no longer" or "exactly"
|
|
26
|
+
only when you mean it: the recorder refuses a check the step's words do
|
|
27
|
+
not name. A containment check reads its value as a whole part of the URL
|
|
28
|
+
and outside the query, unless the value names one (`?`, `=`, `&`).
|
|
29
|
+
|
|
21
30
|
## Scenario Outlines
|
|
22
31
|
|
|
23
32
|
One cache serves every row: `<param>` placeholders stay in the cache and
|