decisium 0.1.2 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +76 -20
- package/dist/cli.js +2 -2
- package/dist/lint.js +2 -2
- package/package.json +3 -4
- package/skills/decisium/SKILL.md +12 -15
- package/docs/cookbook.md +0 -346
package/README.md
CHANGED
|
@@ -1,34 +1,90 @@
|
|
|
1
|
-
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img alt="decisium" src="https://raw.githubusercontent.com/andletenkov/decisium/main/site/assets/logo-black.png#gh-light-mode-only" width="320">
|
|
3
|
+
<img alt="decisium" src="https://raw.githubusercontent.com/andletenkov/decisium/main/site/assets/logo-white.png#gh-dark-mode-only" width="320">
|
|
4
|
+
</p>
|
|
2
5
|
|
|
3
|
-
Semantic UI testing with decision models
|
|
4
|
-
model, picks among the actions the page actually offers, and code decides pass/fail. No selectors, no LLM, no prompts.
|
|
5
|
-
Record/replay cassettes make CI deterministic with zero model calls.
|
|
6
|
+
<p align="center"><b>Semantic UI testing with decision models.</b><br>Plain-English Gherkin, a real browser, pass/fail decided by code.</p>
|
|
6
7
|
|
|
7
|
-
|
|
8
|
-
|
|
8
|
+
<p align="center">
|
|
9
|
+
<a href="https://www.npmjs.com/package/decisium"><img alt="npm" src="https://img.shields.io/npm/v/decisium?color=black"></a>
|
|
10
|
+
<a href="LICENSE"><img alt="MIT" src="https://img.shields.io/npm/l/decisium?color=black"></a>
|
|
11
|
+
<a href="https://andletenkov.github.io/decisium/"><img alt="docs" src="https://img.shields.io/badge/docs-andletenkov.github.io%2Fdecisium-black"></a>
|
|
12
|
+
</p>
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
Write what the user does. Decisium runs it in Playwright:
|
|
17
|
+
|
|
18
|
+
```gherkin
|
|
19
|
+
@url:https://www.saucedemo.com/
|
|
20
|
+
Feature: Checkout
|
|
21
|
+
|
|
22
|
+
Scenario: Checkout requires a first name
|
|
23
|
+
When I log in as "standard_user" with password "$SAUCE_PASSWORD"
|
|
24
|
+
And I add "Sauce Labs Backpack" to the cart
|
|
25
|
+
And I open the cart
|
|
26
|
+
And I start the checkout
|
|
27
|
+
And I try to continue without filling in the form
|
|
28
|
+
Then the error should say "First Name is required"
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
No selectors, no page objects, no step definitions.
|
|
32
|
+
|
|
33
|
+
## How it works
|
|
34
|
+
|
|
35
|
+
Each Step is executed by **Jev**, TypeSafe's decision model. Jev does not generate text or code. It picks among the
|
|
36
|
+
options Decisium enumerates from the page (elements, actions, check kinds) and returns a probability for each.
|
|
37
|
+
**Code, not the model, decides whether a Step passed**: a value check compares the element's value with the quoted one,
|
|
38
|
+
a presence check looks for the text, a state check reads the control's state. No LLM, no prompts, no invented selectors.
|
|
39
|
+
|
|
40
|
+
What each Step resolved to is recorded in a **cassette** next to the feature. CI replays it with **zero model calls**,
|
|
41
|
+
the same outcome every run; a real UI change fails as `UI_CHANGED`.
|
|
9
42
|
|
|
10
43
|
## Install
|
|
11
44
|
|
|
12
45
|
```sh
|
|
13
46
|
npm install --save-dev decisium
|
|
14
47
|
npx playwright install chromium
|
|
15
|
-
|
|
16
|
-
npx decisium doctor
|
|
17
|
-
npx decisium run my.feature
|
|
48
|
+
echo 'TYPESAFE_API_KEY=...' >> .env # live runs only; replays need no key
|
|
49
|
+
npx decisium doctor
|
|
18
50
|
```
|
|
19
51
|
|
|
20
|
-
|
|
52
|
+
Node 22.18+ or 23.6+.
|
|
21
53
|
|
|
22
|
-
##
|
|
54
|
+
## Use
|
|
23
55
|
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
worldParameters: { decisium: { record: "new" } },
|
|
30
|
-
};
|
|
56
|
+
```sh
|
|
57
|
+
npx decisium lint checkout.feature # how each Step will be read; flags known-bad wordings
|
|
58
|
+
npx decisium run checkout.feature # live: asks Jev, records checkout.cassette.json
|
|
59
|
+
npx decisium run checkout.feature --record none # replay: zero model calls (CI)
|
|
60
|
+
npx decisium step "I open the cart" --url https://www.saucedemo.com/ # try one wording
|
|
31
61
|
```
|
|
32
62
|
|
|
33
|
-
|
|
34
|
-
|
|
63
|
+
A failed run prints the reason under each Step and writes `runs/<timestamp>/report.md`: the Step, Jev's answers, the
|
|
64
|
+
page text around the element, a screenshot, a Playwright trace and a rerun command, ready for you or a coding agent.
|
|
65
|
+
|
|
66
|
+
Decisium runs inside cucumber-js: your `cucumber.mjs`, profiles, formatters (`--format junit:junit.xml`) and your own
|
|
67
|
+
step definitions (`--import`, with `this.page` and `this.context`) keep working.
|
|
68
|
+
|
|
69
|
+
## Docs
|
|
70
|
+
|
|
71
|
+
**[andletenkov.github.io/decisium](https://andletenkov.github.io/decisium/)**
|
|
72
|
+
|
|
73
|
+
- [Quick start](https://andletenkov.github.io/decisium/quick-start.html)
|
|
74
|
+
- [Writing features](https://andletenkov.github.io/decisium/writing-features.html)
|
|
75
|
+
- [Writing Steps](https://andletenkov.github.io/decisium/steps.html): every kind of action and check, and how to word it
|
|
76
|
+
- [Client steps](https://andletenkov.github.io/decisium/client-steps.html) · [Browser context](https://andletenkov.github.io/decisium/browser-context.html) · [Network replay](https://andletenkov.github.io/decisium/network-replay.html)
|
|
77
|
+
- [Record and replay](https://andletenkov.github.io/decisium/record-replay.html) · [CI](https://andletenkov.github.io/decisium/ci.html) · [Configuration](https://andletenkov.github.io/decisium/configuration.html)
|
|
78
|
+
- [Examples](https://andletenkov.github.io/decisium/examples.html) against real public sites
|
|
79
|
+
- An agent skill for coding agents ships in the package: `node_modules/decisium/skills/decisium/SKILL.md`
|
|
80
|
+
|
|
81
|
+
## Jev and your data
|
|
82
|
+
|
|
83
|
+
Live runs call Jev through the TypeSafe API and need a TypeSafe API key; Jev is TypeSafe's hosted service under its own
|
|
84
|
+
terms. Per Turn it receives the Step's text and a text snapshot of the page (the accessibility tree: roles, names,
|
|
85
|
+
visible text, link URLs). No screenshots, cookies or network traffic. Write credentials as `"$NAME"`: only the token
|
|
86
|
+
leaves your machine. Replays send nothing.
|
|
87
|
+
|
|
88
|
+
## License
|
|
89
|
+
|
|
90
|
+
[MIT](LICENSE) © Andrei Letenkov
|
package/dist/cli.js
CHANGED
|
@@ -32,13 +32,13 @@ the scenario (or Examples row) at that line.
|
|
|
32
32
|
doctor: checks Node, Chromium and TYPESAFE_API_KEY (one tiny Jev call); given features, also opens
|
|
33
33
|
every start URL.
|
|
34
34
|
lint: no browser, no Jev. Per Step: its kind, the quoted values, keys, secret placeholders, and warnings for wordings
|
|
35
|
-
docs
|
|
35
|
+
the docs mark as ones to avoid (andletenkov.github.io/decisium/steps.html).
|
|
36
36
|
Configuration is cucumber-js's: run loads your cucumber.{js,mjs,cjs,json,yaml} (and -p profiles); --format, --import and
|
|
37
37
|
-p (repeatable) are cucumber's flags, added to it. --format name[:path]: cucumber's junit, json, html, message or any
|
|
38
38
|
formatter package, e.g. --format junit:junit.xml or --format cucumberjs-qase-reporter:/dev/null. Only one formatter may
|
|
39
39
|
write to stdout: one configured without a path takes it, else Decisium's console output does.
|
|
40
40
|
--import: your own cucumber step definitions or hooks. A line one of them matches runs it, not Jev; the World gives them
|
|
41
|
-
this.page (the current tab) and this.context (Playwright). See
|
|
41
|
+
this.page (the current tab) and this.context (Playwright). See andletenkov.github.io/decisium/client-steps.html.
|
|
42
42
|
Options in the configuration: worldParameters: { decisium: { url, record, headed, judged, strict, verbose, debug,
|
|
43
43
|
agentReport } }; flags and DECISIUM_* env (DECISIUM_URL, DECISIUM_RECORD, DECISIUM_STRICT=1, ...) override them.
|
|
44
44
|
-v prints every Observation and Jev answer (always saved to runs/<timestamp>/log.txt); --debug shows stack traces.
|
package/dist/lint.js
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
// `decisium lint <file.feature>`: what each Step will be taken as, read without a browser or Jev (task 0031).
|
|
2
2
|
// Per Step: its kind, the quoted values Jev can pick from, the keys it presses, secret placeholders, and warnings for wordings that
|
|
3
|
-
// docs/
|
|
3
|
+
// the docs mark as ones to avoid (docs/site/steps.md).
|
|
4
4
|
import { readFileSync } from "node:fs";
|
|
5
5
|
import { parseFeature } from "./feature.js";
|
|
6
6
|
import { KIND } from "./report.js";
|
|
7
7
|
import { placeholder, readStep } from "./step.js";
|
|
8
|
-
const SEE = "see
|
|
8
|
+
const SEE = "see https://andletenkov.github.io/decisium/steps.html";
|
|
9
9
|
/** Cookbook warnings for one Step. `before`: the scenario's earlier Steps. */
|
|
10
10
|
export function warnings(step, before, env = process.env) {
|
|
11
11
|
const t = step.text;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "decisium",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.3",
|
|
4
4
|
"description": "Semantic UI testing with decision models: plain-English Gherkin Steps executed by Jev in a real browser, pass/fail decided by code, record/replay cassettes",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -43,17 +43,16 @@
|
|
|
43
43
|
"files": [
|
|
44
44
|
"dist",
|
|
45
45
|
"skills",
|
|
46
|
-
"docs/cookbook.md",
|
|
47
46
|
"docs/configuration.md"
|
|
48
47
|
],
|
|
49
48
|
"scripts": {
|
|
50
49
|
"test": "node --env-file-if-exists=.env --test test/*.test.ts",
|
|
51
50
|
"typecheck": "tsc -p .",
|
|
52
|
-
"example:server": "node
|
|
53
|
-
"cookbook": "node scripts/cookbook.ts",
|
|
51
|
+
"example:server": "node test/fixtures/campaign-manager/server.ts",
|
|
54
52
|
"build": "rm -rf dist && tsc -p tsconfig.build.json",
|
|
55
53
|
"prepublishOnly": "npm run build",
|
|
56
54
|
"site": "node scripts/site.ts",
|
|
55
|
+
"version": "node scripts/site.ts && git add site",
|
|
57
56
|
"pack:smoke": "scripts/pack-smoke.sh"
|
|
58
57
|
},
|
|
59
58
|
"dependencies": {
|
package/skills/decisium/SKILL.md
CHANGED
|
@@ -3,32 +3,29 @@ name: decisium
|
|
|
3
3
|
description: Write or repair Decisium Gherkin features (natural-language Steps run by Jev). Use when writing a .feature for Decisium, or when a Step ends AMBIGUOUS, UNVERIFIABLE or STEP_TIMEOUT.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
Decisium runs each Step by asking Jev, a model that only picks among offered answers
|
|
7
|
-
Step's wording decides most outcomes
|
|
8
|
-
|
|
9
|
-
the rules; this skill is the procedure.
|
|
6
|
+
Decisium runs each Step by asking Jev, a decision model that only picks among offered answers; code decides pass/fail.
|
|
7
|
+
A Step's wording decides most outcomes. The wording rules, per kind of Step, are in
|
|
8
|
+
https://andletenkov.github.io/decisium/steps.html. Read it before writing a Step. This skill is the procedure.
|
|
10
9
|
|
|
11
10
|
## Write
|
|
12
11
|
|
|
13
12
|
1. Write the feature: an `@url:<start url>` tag, then one Scenario per user journey, each starting from a fresh browser.
|
|
14
|
-
|
|
13
|
+
Word each Step like a **Write** row of its kind (navigate, type, select, act, attempt, keys, value / presence /
|
|
15
14
|
scoped / state check, memory, secrets). Quote every value the Step uses and every value a check compares.
|
|
16
15
|
Credentials go in as `"$NAME"` placeholders, with the variables set in the environment or `.env`.
|
|
17
|
-
2. Run `
|
|
18
|
-
|
|
19
|
-
purpose with a `#` comment saying why (for example, a scenario meant to fail).
|
|
16
|
+
2. Run `npx decisium lint <file.feature>`. It shows how each Step will be read (kind, quoted values, placeholders) and
|
|
17
|
+
flags wordings to avoid with ⚠. Done when every ⚠ is reworded away, or kept on purpose with a `#` comment saying why.
|
|
20
18
|
|
|
21
19
|
## Run and repair
|
|
22
20
|
|
|
23
|
-
3. Run `
|
|
24
|
-
4. On a failure, read `runs/<id>/report.md`: the Step at `file:line`, Jev's answers
|
|
25
|
-
|
|
21
|
+
3. Run `npx decisium run <file.feature>` (live Jev the first time; it records a cassette next to the feature).
|
|
22
|
+
4. On a failure, read `runs/<id>/report.md`: the Step at `file:line`, Jev's answers, the page text around the element,
|
|
23
|
+
and a rerun command for that scenario. Classify each failure:
|
|
26
24
|
- **The app is wrong** (ASSERTION_FAILED, BLOCKED with the expected control truly missing): keep the Step, and
|
|
27
25
|
report the app behaviour.
|
|
28
|
-
- **The wording is unclear** (AMBIGUOUS, UNVERIFIABLE, a check that picked the wrong element): reword using
|
|
29
|
-
|
|
30
|
-
- A **marginal pass** warning means the pass rested on a barely-confident answer. Treat it like an unclear
|
|
31
|
-
wording.
|
|
26
|
+
- **The wording is unclear** (AMBIGUOUS, UNVERIFIABLE, a check that picked the wrong element): reword using
|
|
27
|
+
https://andletenkov.github.io/decisium/troubleshooting.html, then rerun just that scenario.
|
|
28
|
+
- A **marginal pass** warning means the pass rested on a barely-confident answer. Treat it like an unclear wording.
|
|
32
29
|
5. Keep the test's intent fixed while rewording. The quoted expected values and the area a check looks in stay as
|
|
33
30
|
written. A rewording that makes a check pass by reading a different element (a link instead of the heading, a
|
|
34
31
|
search box instead of the article) is a false pass. Compare Jev's subject in `report.md` with what the scenario
|
package/docs/cookbook.md
DELETED
|
@@ -1,346 +0,0 @@
|
|
|
1
|
-
# Step cookbook
|
|
2
|
-
|
|
3
|
-
Step wordings checked against Jev. Every row links to a Step in `examples/` and gives the outcome that Step had in the
|
|
4
|
-
last replay. `node scripts/cookbook.ts` replays the examples from their cassettes (no Jev calls) and fails when a
|
|
5
|
-
linked Step is missing, is worded differently, or ended another way. Rows marked ✗ are wordings to avoid. Each one
|
|
6
|
-
says what went wrong and links to the wording that works.
|
|
7
|
-
|
|
8
|
-
`decisium lint <file.feature>` reads a feature without a browser or Jev and flags the wordings below.
|
|
9
|
-
|
|
10
|
-
## Rules of thumb
|
|
11
|
-
|
|
12
|
-
1. **One Step, one task**, named by what the user wants, not by the clicks: "I save the draft", not "I click the
|
|
13
|
-
button labelled Save draft".
|
|
14
|
-
2. **Quote every value**: what to type, the option to pick, the text a check expects. Jev picks among the quoted
|
|
15
|
-
values and never invents one. Numbers with a unit can go unquoted ("2 weeks").
|
|
16
|
-
3. **A check quotes exactly what code compares.** A value check quotes one value, and a presence check quotes the
|
|
17
|
-
text to look for. A check that quotes nothing is UNVERIFIABLE unless it refers back to a value an earlier Step
|
|
18
|
-
quoted.
|
|
19
|
-
4. **Name things the way the page labels them**: "heading", "error", "status", "the calendar". Avoid words that
|
|
20
|
-
also name something else on the page. "Title" can mean the browser tab's title.
|
|
21
|
-
5. **Say "open" when you mean open.** "Search for" does a search. It does not open a result.
|
|
22
|
-
6. **Write `"$NAME"` for secrets.** The value comes from the environment and Jev never sees it.
|
|
23
|
-
7. **No CSS/XPath selectors in Steps.** Quote what the page shows instead (`the "Search page" link`). When no
|
|
24
|
-
wording works, write a client step (below), not a selector in the Step.
|
|
25
|
-
|
|
26
|
-
## Flaky Steps
|
|
27
|
-
|
|
28
|
-
A live run asks Jev every time; a Step whose answer is close to the gate can end differently from run to run.
|
|
29
|
-
|
|
30
|
-
1. **CI replays, it does not ask.** Record once (`--record new` or `all`), check the outcomes, commit the cassettes,
|
|
31
|
-
and run CI with `--record none`: every Step replays its recorded resolution, 0 Jev calls, the same outcome each run.
|
|
32
|
-
A real UI change fails as UI_CHANGED; heal it with `--record new`. Live flakiness only matters when recording.
|
|
33
|
-
2. **Read the reason, then reword.** The AMBIGUOUS reason names what Jev split between. Quote the one you mean:
|
|
34
|
-
`I leave for the search page` split the "Search page" link and the nav's "Search" (51% / 49%, DONE 1/3);
|
|
35
|
-
`I leave by the "Search page" link` was DONE 3/3 ([dialogs.feature:24](../examples/cookbook/dialogs.feature#L24)).
|
|
36
|
-
When it splits on *how* (a Search button vs Enter), say how: `... and press Enter`.
|
|
37
|
-
3. **A Step that does two things the page needs anyway is one Step.** On OrangeHRM the role filter applies only when
|
|
38
|
-
Search is clicked, so `I filter the users by user role "Admin"` clicks it; a following `I search` has nothing
|
|
39
|
-
left to do and goes wrong. Do not split what the page cannot do separately.
|
|
40
|
-
4. **Still no stable wording: a client step.** Code in `<feature>.steps.ts` can use any Playwright locator (see
|
|
41
|
-
[Client steps](#client-steps-browser-context-and-network)). Each one marks a gap worth reporting.
|
|
42
|
-
|
|
43
|
-
Why not selectors: they tie the test to markup and bypass Jev, and many flaky Steps are not about the target but
|
|
44
|
-
about whether the Step is done, which a selector does not fix.
|
|
45
|
-
|
|
46
|
-
## Navigate
|
|
47
|
-
|
|
48
|
-
| | Wording | Outcome | Example | Why |
|
|
49
|
-
|---|---|---|---|---|
|
|
50
|
-
| ✓ | `I open the calendar` | DONE | [campaigns.feature:15](../examples/campaign-manager/campaigns.feature#L15) | Names the place, not the link. |
|
|
51
|
-
| ✓ | `I open the "Spring Sale" campaign` | DONE | [campaigns.feature:20](../examples/campaign-manager/campaigns.feature#L20) | Quoted name picks the row. |
|
|
52
|
-
| ✓ | `I start a new campaign` | DONE | [campaigns.feature:6](../examples/campaign-manager/campaigns.feature#L6) | The task, not the button text. |
|
|
53
|
-
| ✓ | `I open the "Ada Lovelace" article through the search` | DONE | [lab.feature:6](../examples/cookbook/lab.feature#L6) | Says how *and* that it opens: Jev types, submits, then clicks the result. |
|
|
54
|
-
| ✗ | `I search for "Ada Lovelace"` | DONE | [lab.feature:17](../examples/cookbook/lab.feature#L17) | Done once the search ran, so the result is never opened. The check after it is AMBIGUOUS ([lab.feature:18](../examples/cookbook/lab.feature#L18)). Before the heading wording, the check passed on the result *link* named "Ada Lovelace", a false pass that only the marginal-pass warning caught. |
|
|
55
|
-
| ✓ | `I go to "?keys"` | DONE | [navigation.feature:6](../examples/cookbook/navigation.feature#L6) | A path, query, `#hash` or full URL (quoted or not) resolves against the start URL and is opened by code, without Jev. A page name ("I go to the Settings page") is Jev's. |
|
|
56
|
-
| ✓ | `I go back` | DONE | [navigation.feature:11](../examples/cookbook/navigation.feature#L11) | Code, without Jev. So are `I go forward` and `I reload the page` ([navigation.feature:21](../examples/cookbook/navigation.feature#L21)). |
|
|
57
|
-
| ✓ | `I return to the previous page` | DONE | [navigation.feature:16](../examples/cookbook/navigation.feature#L16) | Other wordings: Jev is offered back, forward and reload every Turn and picked back (0.91). |
|
|
58
|
-
| ✓ | `I open the contacts` | DONE | [navigation.feature:25](../examples/cookbook/navigation.feature#L25) | A link that opens a new tab: the Steps after it act in the new tab. The Observation shows `tabs value="2 open, this is tab 2"`. |
|
|
59
|
-
| ✓ | `I close the tab` | DONE | [navigation.feature:27](../examples/cookbook/navigation.feature#L27) | Code, without Jev; back on the previous tab. So is `I switch back to the previous tab`. Inside a longer Step Jev is offered "close tab" and "switch tab", but after closing it barely takes the Step as done (0.24-0.37, BLOCKED live), so say it as its own Step. |
|
|
60
|
-
|
|
61
|
-
## Type
|
|
62
|
-
|
|
63
|
-
| | Wording | Outcome | Example | Why |
|
|
64
|
-
|---|---|---|---|---|
|
|
65
|
-
| ✓ | `I enter "Winter Sale" as the campaign name` | DONE | [campaigns.feature:7](../examples/campaign-manager/campaigns.feature#L7) | Value quoted, field named. |
|
|
66
|
-
| ✓ | `I set the activation date to "2026-12-01"` | DONE | [campaigns.feature:8](../examples/campaign-manager/campaigns.feature#L8) | Dates in the format the field takes. |
|
|
67
|
-
| ✓ | `I search for "Ada" and submit the search` | DONE | [lab.feature:21](../examples/cookbook/lab.feature#L21) | "and submit" makes typing alone not enough. Without it, Jev once judged `I search for "Ada"` done right after typing. |
|
|
68
|
-
|
|
69
|
-
## Select
|
|
70
|
-
|
|
71
|
-
| | Wording | Outcome | Example | Why |
|
|
72
|
-
|---|---|---|---|---|
|
|
73
|
-
| ✓ | `I set the activation period to 2 weeks` | DONE | [campaigns.feature:9](../examples/campaign-manager/campaigns.feature#L9) | A number with a unit counts as a value. The native select's options are the candidates. |
|
|
74
|
-
|
|
75
|
-
## Submit and act
|
|
76
|
-
|
|
77
|
-
| | Wording | Outcome | Example | Why |
|
|
78
|
-
|---|---|---|---|---|
|
|
79
|
-
| ✓ | `I save the draft` | DONE | [campaigns.feature:10](../examples/campaign-manager/campaigns.feature#L10) | |
|
|
80
|
-
| ✓ | `I schedule the campaign` | DONE | [campaigns.feature:12](../examples/campaign-manager/campaigns.feature#L12) | |
|
|
81
|
-
| ✓ | `I delete the campaign` | DONE | [campaigns.feature:29](../examples/campaign-manager/campaigns.feature#L29) | Jev opens the "More actions" menu first. Several actions in one Step are fine when they serve one task. |
|
|
82
|
-
| ✓ | `I uncheck "Newsletter"` | DONE | [lab.feature:42](../examples/cookbook/lab.feature#L42) | Code reads the box after the click: done once it is unchecked. |
|
|
83
|
-
| ✓ | `I check "Weekly digest"` | DONE | [lab.feature:48](../examples/cookbook/lab.feature#L48) | Only when the Step does nothing else: with "and ..." Jev decides, and it kept clicking the box (task 0033). |
|
|
84
|
-
|
|
85
|
-
## Keys
|
|
86
|
-
|
|
87
|
-
A key is pressed only when the Step names it after "press" or "hit", quoted or not: Playwright's names (`Escape`,
|
|
88
|
-
`Tab`, `ArrowDown`, `PageDown`, `Home`, `F2`...), aliases (`Esc`, `down arrow`, `Return`, `Space`) and `+`
|
|
89
|
-
combinations (`Ctrl+K`, `Shift+Tab`; `Ctrl` is Control, `Cmd` is Meta, pressed as written). The key goes to whatever
|
|
90
|
-
has focus. A Step that is only "press X" is pressed by code without Jev and is DONE; the checks after it verify the
|
|
91
|
-
effect. Inside a longer Step, Jev is offered just the keys it names. `lint` lists the keys and warns on a name that is
|
|
92
|
-
not a key.
|
|
93
|
-
|
|
94
|
-
| | Wording | Outcome | Example | Why |
|
|
95
|
-
|---|---|---|---|---|
|
|
96
|
-
| ✓ | `I press Escape` | DONE | [keys.feature:8](../examples/cookbook/keys.feature#L8) | Only a key: code presses it. The check after it reads the effect ([keys.feature:9](../examples/cookbook/keys.feature#L9)). |
|
|
97
|
-
| ✓ | `I open the command palette by pressing Ctrl+K` | DONE | [keys.feature:12](../examples/cookbook/keys.feature#L12) | Jev is offered "press Control+K" next to the page's elements, and picked it (96%). |
|
|
98
|
-
| ✓ | `I press the down arrow` | DONE | [keys.feature:16](../examples/cookbook/keys.feature#L16) | Alias for `ArrowDown`, sent to the focused list. |
|
|
99
|
-
| ✓ | `I press "Tab"` | DONE | [keys.feature:21](../examples/cookbook/keys.feature#L21) | Quoting is optional. |
|
|
100
|
-
| ✗ | `I close the help with the Escape key` | BLOCKED | [keys.feature:27](../examples/cookbook/keys.feature#L27) | No "press" or "hit", so no key is offered, and the help has no close button (live runs also ended AMBIGUOUS). Say `I press Escape`. |
|
|
101
|
-
|
|
102
|
-
## Hover
|
|
103
|
-
|
|
104
|
-
| | Wording | Outcome | Example | Why |
|
|
105
|
-
|---|---|---|---|---|
|
|
106
|
-
| ✗ | `I open the Products menu` | BLOCKED | [hover.feature:6](../examples/cookbook/hover.feature#L6) | A menu that opens only on mouse-over, and a click closes it. Jev clicks Products with any wording (0.71-0.80; accepted in task 0065 decision 9), the menu stays closed, and after two clicks with no visible change the Step is BLOCKED. Before task 0067 it was a false DONE: the button's new focus counted as a change. |
|
|
107
|
-
| ✓ | `I hover over the help button` | DONE | [hover.feature:11](../examples/cookbook/hover.feature#L11) | Saying it is surer (hover 0.75-0.81). A tooltip it shows is in the page like any text. |
|
|
108
|
-
| ✓ | `the tooltip should say "Opens the docs"` | PASSED | [hover.feature:12](../examples/cookbook/hover.feature#L12) | `the tooltip should be ...` was a marginal pass (subject 0.89 vs the button it describes). |
|
|
109
|
-
| ✓ | `the Save button's tooltip should be "Save your work"` | PASSED | [hover.feature:15](../examples/cookbook/hover.feature#L15) | A `title` attribute is the element's description (`description="..."`), compared when the control has no value. No hover needed. |
|
|
110
|
-
|
|
111
|
-
## Radios and toggles
|
|
112
|
-
|
|
113
|
-
| | Wording | Outcome | Example | Why |
|
|
114
|
-
|---|---|---|---|---|
|
|
115
|
-
| ✓ | `I accept the terms` | DONE | [toggles.feature:6](../examples/cookbook/toggles.feature#L6) | A Step that clicks a checkbox, switch or radio wants it on, no check word needed; code reads the box after the click. |
|
|
116
|
-
| ✓ | `I turn off dark mode` | DONE | [toggles.feature:7](../examples/cookbook/toggles.feature#L7) | A negation (disable, turn off, decline, reject, opt out, unsubscribe) wants it off. Works for `role=switch`. |
|
|
117
|
-
| ✓ | `I choose Express shipping` | DONE | [toggles.feature:8](../examples/cookbook/toggles.feature#L8) | A radio is done once the clicked one is `[checked]`. |
|
|
118
|
-
| ✓ | `I subscribe to the newsletter` | DONE | [toggles.feature:9](../examples/cookbook/toggles.feature#L9) | Already on: DONE without a click. Jev wavers between nothing to do and the box (none 0.52, box 0.37); code counts both as done. |
|
|
119
|
-
|
|
120
|
-
## Files
|
|
121
|
-
|
|
122
|
-
| | Wording | Outcome | Example | Why |
|
|
123
|
-
|---|---|---|---|---|
|
|
124
|
-
| ✓ | `I upload "notes.txt" with the Attach file button` | DONE | [files.feature:6](../examples/cookbook/files.feature#L6) | "upload" or "attach" plus a quoted path (relative to the feature file): the button Jev picks is clicked and the file chooser it opens gets the file, so a hidden `<input type=file>` works. Only a quoted path is uploaded; one that is no file fails ACTION_FAILED. |
|
|
125
|
-
| ✓ | `I export the contacts as CSV` | DONE | [files.feature:10](../examples/cookbook/files.feature#L10) | A download is saved to the scenario's run folder. |
|
|
126
|
-
| ✓ | `the downloaded file name should be "contacts.csv"` | PASSED | [files.feature:11](../examples/cookbook/files.feature#L11) | The last download is readable as `download "downloaded file"`: its name, then a text file's contents. `should be named` was a marginal pass (kind value 0.80-0.86). An empty file fails. |
|
|
127
|
-
| ✓ | `the downloaded file should contain "Ada Lovelace,engineer"` | PASSED | [files.feature:12](../examples/cookbook/files.feature#L12) | Text and CSV contents are compared like any value; binary files only by name. |
|
|
128
|
-
|
|
129
|
-
## Double-click, right-click, drag and sliders
|
|
130
|
-
|
|
131
|
-
| | Wording | Outcome | Example | Why |
|
|
132
|
-
|---|---|---|---|---|
|
|
133
|
-
| ✓ | `I double-click the note title and change it to "Groceries"` | DONE | [gestures.feature:6](../examples/cookbook/gestures.feature#L6) | With every target Jev answers how to use it: as usual, hover, double click, right click or drag. Say the edit in the same Step: `I double-click the note title to edit it` alone was never done (done 0.64-0.73), Jev kept double-clicking the field. |
|
|
134
|
-
| ✓ | `I right-click report.pdf and choose "Delete"` | DONE | [gestures.feature:10](../examples/cookbook/gestures.feature#L10) | A custom context menu is in the next Observation. A plain element is offered only with a pointer cursor (`[cursor=pointer]`) or `draggable`. |
|
|
135
|
-
| ✓ | `I drag "Fix bug" to the Done list` | DONE | [gestures.feature:14](../examples/cookbook/gestures.feature#L14) | Jev then picks where to drop it among readable elements and areas. Done came at 0.75-0.9: 1 of 7 live runs ended AMBIGUOUS after the drag. |
|
|
136
|
-
| ✓ | `"Fix bug" should appear in the Done list` | PASSED | [gestures.feature:15](../examples/cookbook/gestures.feature#L15) | `the Done list should contain "Fix bug"` was AMBIGUOUS (kind value vs presence). |
|
|
137
|
-
| ✓ | `I set the volume to "75"` | DONE | [gestures.feature:19](../examples/cookbook/gestures.feature#L19) | A range input is a slider: it is filled with the quoted value like a text field. |
|
|
138
|
-
|
|
139
|
-
## Scroll inside lists and panels, and up
|
|
140
|
-
|
|
141
|
-
| | Wording | Outcome | Example | Why |
|
|
142
|
-
|---|---|---|---|---|
|
|
143
|
-
| ✓ | `I scroll down the Inbox until "Message 15" is shown` | DONE | [scroll.feature:6](../examples/cookbook/scroll.feature#L6) | A list or panel that scrolls on its own is marked `[scrollable]` and offered as a target; Jev answers the gesture `scroll down`, the mouse wheel turns over it. Content already rendered inside a panel is in the snapshot without scrolling: only lazy-loaded content needs it. |
|
|
144
|
-
| ✓ | `"Message 15" should appear in the Inbox` | PASSED | [scroll.feature:7](../examples/cookbook/scroll.feature#L7) | |
|
|
145
|
-
| ✓ | `I scroll up the Chat until "Chat line 5" is shown` | DONE | [scroll.feature:10](../examples/cookbook/scroll.feature#L10) | Gesture `scroll up` (older messages loaded at the top). The page itself has page actions `scroll` (down) and `scroll up`. |
|
|
146
|
-
| ✓ | `"Chat line 5" should appear in the Chat` | PASSED | [scroll.feature:11](../examples/cookbook/scroll.feature#L11) | |
|
|
147
|
-
|
|
148
|
-
## iframes and web components
|
|
149
|
-
|
|
150
|
-
Elements inside iframes and open shadow roots are in the Observation like any other: no frame or component to name in
|
|
151
|
-
the Step. Cross-origin iframes (another port or host, out of process) worked the same in a local check; not covered by
|
|
152
|
-
an example. Closed shadow roots (`attachShadow({ mode: "closed" })`) are invisible to the browser's accessibility
|
|
153
|
-
snapshot and out of scope.
|
|
154
|
-
|
|
155
|
-
| | Wording | Outcome | Example | Why |
|
|
156
|
-
|---|---|---|---|---|
|
|
157
|
-
| ✓ | `I enter "ada@example.com" in Email` | DONE | [frames.feature:6](../examples/cookbook/frames.feature#L6) | The Email field is inside a same-origin iframe. |
|
|
158
|
-
| ✓ | `I click Subscribe` | DONE | [frames.feature:7](../examples/cookbook/frames.feature#L7) | |
|
|
159
|
-
| ✓ | `the newsletter status should be "Subscribed ada@example.com"` | PASSED | [frames.feature:8](../examples/cookbook/frames.feature#L8) | Checks read text inside the iframe. |
|
|
160
|
-
| ✓ | `the Email field should be "ada@example.com"` | PASSED | [frames.feature:9](../examples/cookbook/frames.feature#L9) | |
|
|
161
|
-
| ✓ | `I click Like` | DONE | [frames.feature:12](../examples/cookbook/frames.feature#L12) | The Like button is inside a web component's open shadow root. |
|
|
162
|
-
| ✓ | `the Like button should be pressed` | PASSED | [frames.feature:13](../examples/cookbook/frames.feature#L13) | |
|
|
163
|
-
| ✓ | `"1 like" should be shown` | PASSED | [frames.feature:14](../examples/cookbook/frames.feature#L14) | |
|
|
164
|
-
|
|
165
|
-
## Client steps, browser context and network
|
|
166
|
-
|
|
167
|
-
Your own cucumber step definitions run beside Decisium: a line one of them matches runs your code, every other line
|
|
168
|
-
goes to Jev. Load them like in any cucumber project: `import` in your cucumber configuration, or
|
|
169
|
-
`decisium run <file.feature> --import <steps.ts>` (repeatable; see [configuration](configuration.md)). The World gives them
|
|
170
|
-
`this.page` (the current tab; a popup the app opens becomes it) and `this.context` (the Playwright `BrowserContext`).
|
|
171
|
-
They run after the start page has opened, so a mock covers requests made from then on (e.g. the next click).
|
|
172
|
-
|
|
173
|
-
```ts
|
|
174
|
-
// examples/cookbook/network.steps.ts
|
|
175
|
-
import { Given } from "@cucumber/cucumber";
|
|
176
|
-
import type { BrowserContext } from "playwright";
|
|
177
|
-
|
|
178
|
-
Given("the campaigns API returns {string}", async function (this: { context: BrowserContext }, name: string) {
|
|
179
|
-
await this.context.route("**/api/campaigns", route =>
|
|
180
|
-
route.fulfill({ json: [{ id: 1, name, status: "Draft" }], headers: { "access-control-allow-origin": "*" } }));
|
|
181
|
-
});
|
|
182
|
-
```
|
|
183
|
-
|
|
184
|
-
Recipes, each a client step:
|
|
185
|
-
|
|
186
|
-
- **Mock a request**: `this.context.route(glob, route => route.fulfill({ json }))`, as above.
|
|
187
|
-
- **Cookies**: `this.context.addCookies([{ name, value, url }])`, then `this.page.reload()` if the page already read them.
|
|
188
|
-
- **Storage**: `this.page.evaluate(([k, v]) => localStorage.setItem(k, v), [key, value])`, then `this.page.reload()`.
|
|
189
|
-
- **Log in through the API**: `const r = await this.context.request.post(url, { data })` shares cookies with the
|
|
190
|
-
page; then `this.page.goto(...)` or reload. Faster than typing the form in every scenario.
|
|
191
|
-
- **Permissions**: `this.context.grantPermissions(["clipboard-read"])`.
|
|
192
|
-
- **Several users**: `const other = await this.context.browser()!.newContext()` and `await other.newPage()` for a
|
|
193
|
-
second user's session; Decisium's Steps act on `this.page` only, so check the other user's page in client code.
|
|
194
|
-
- **Wait for a request**: nothing to write. An Action step ends only once the requests it started have finished
|
|
195
|
-
(up to 5 s), so the next check sees their result.
|
|
196
|
-
|
|
197
|
-
Tags set browser context options before the page opens (the last tag wins, scenario tags over feature tags):
|
|
198
|
-
`@viewport:390x844` or a Playwright device name with dashes for spaces (`@viewport:iPhone-13`: size, user agent,
|
|
199
|
-
touch), `@locale:de-DE`, `@timezone:Europe/Berlin`, `@geolocation:52.52,13.40` (also grants the permission).
|
|
200
|
-
|
|
201
|
-
`@replay-network:<glob>` records the browser requests matching the glob and replays them, following `--record` like
|
|
202
|
-
the cassette: `all` records, `none` replays, `new` replays a recording and records a missing one. The recording is
|
|
203
|
-
`<feature>.network/<scenario>.har` next to the cassette (bodies in files beside it). Replay matches method + URL + the
|
|
204
|
-
n-th call to that URL, not the request body, so prompts with timestamps or ids still replay. A matching request that
|
|
205
|
-
is not in the recording fails the Step as "network not in recording" and never goes live. Use it to stop paying for an
|
|
206
|
-
app's expensive backend calls (agent/LLM requests). Limits: a streamed response (SSE) replays in one chunk, so the
|
|
207
|
-
final text shows at once; WebSockets are not recorded; requests the app's backend makes cannot be intercepted, only the
|
|
208
|
-
browser's.
|
|
209
|
-
|
|
210
|
-
| | Wording | Outcome | Example | Why |
|
|
211
|
-
|---|---|---|---|---|
|
|
212
|
-
| ✓ | `the Viewport should be "390x844"` | PASSED | [network.feature:8](../examples/cookbook/network.feature#L8) | `@viewport:390x844` sized the page before it opened. |
|
|
213
|
-
| ✓ | `the Locale should be "de-DE"` | PASSED | [network.feature:9](../examples/cookbook/network.feature#L9) | `@locale:de-DE`. |
|
|
214
|
-
| ✓ | `"Mocked Sale" should be shown` | PASSED | [network.feature:14](../examples/cookbook/network.feature#L14) | The click after the client step `the campaigns API returns "Mocked Sale"` (line 12: your code, no Jev call, not in the cassette) got the mocked response. |
|
|
215
|
-
| ✓ | `"Autumn Sale" should be shown` | PASSED | [network.feature:20](../examples/cookbook/network.feature#L20) | `@replay-network:**/api/campaigns`: replayed with the server down. |
|
|
216
|
-
|
|
217
|
-
Out of scope: canvas drawing, maps, captchas, 2FA codes and closed shadow roots. Steps cannot see or drive them; use a
|
|
218
|
-
test mode in the app or a client step (e.g. a 2FA secret to compute the code) where you can.
|
|
219
|
-
|
|
220
|
-
## Clipboard
|
|
221
|
-
|
|
222
|
-
| | Wording | Outcome | Example | Why |
|
|
223
|
-
|---|---|---|---|---|
|
|
224
|
-
| ✓ | `I copy the invite link` | DONE | [clipboard.feature:6](../examples/cookbook/clipboard.feature#L6) | The copy button is clicked like any button; the browser may read and write the clipboard. |
|
|
225
|
-
| ✓ | `the copied text should be "https://example.com/invite/7f3a"` | PASSED | [clipboard.feature:7](../examples/cookbook/clipboard.feature#L7) | Text copied during the scenario is readable as `clipboard "copied text"`, secrets masked. |
|
|
226
|
-
| ✓ | `I paste into the message box` | DONE | [clipboard.feature:8](../examples/cookbook/clipboard.feature#L8) | "paste" without a quoted value: the field Jev picks is clicked and gets Ctrl/Cmd+V. With a quoted value it is typed instead. |
|
|
227
|
-
| ✓ | `the message box should show "https://example.com/invite/7f3a"` | PASSED | [clipboard.feature:9](../examples/cookbook/clipboard.feature#L9) | The pasted text is the field's value. |
|
|
228
|
-
|
|
229
|
-
## Browser dialogs
|
|
230
|
-
|
|
231
|
-
`alert`, `confirm`, `prompt` and "Leave site?" (`beforeunload`) are answered by the Step that opens them, in a later
|
|
232
|
-
Turn: Jev chooses from the dialog's own answers, not the page. OK (or leaving the page) only when the Step clearly means
|
|
233
|
-
it and Jev is sure; otherwise Cancel. An alert is just closed. A prompt gets one of the Step's quoted values. A dialog
|
|
234
|
-
still open when the Step ends is dismissed. Until the next action Step, the last dialog's message is readable for checks
|
|
235
|
-
("the alert should say ..."); what OK or Cancel did is checked on the page.
|
|
236
|
-
|
|
237
|
-
| | Wording | Outcome | Example | Why |
|
|
238
|
-
|---|---|---|---|---|
|
|
239
|
-
| ✓ | `I delete the "Groceries" note` | DONE | [dialogs.feature:6](../examples/cookbook/dialogs.feature#L6) | Jev clicks Delete, then answers the confirm with OK (100%): the Step asks for the deletion. |
|
|
240
|
-
| ✓ | `the confirm should say "Delete the Groceries note?"` | PASSED | [dialogs.feature:8](../examples/cookbook/dialogs.feature#L8) | The last dialog's message is a value check's subject. |
|
|
241
|
-
| ✓ | `I start deleting the "Ideas" note but cancel` | DONE | [dialogs.feature:11](../examples/cookbook/dialogs.feature#L11) | Says to cancel: Cancel (100%). The check after it reads the page ([dialogs.feature:12](../examples/cookbook/dialogs.feature#L12)). |
|
|
242
|
-
| ✓ | `I save the settings` | DONE | [dialogs.feature:15](../examples/cookbook/dialogs.feature#L15) | An alert has only one answer: closed by code, Jev is not asked. |
|
|
243
|
-
| ✓ | `the alert should say "Settings saved"` | PASSED | [dialogs.feature:16](../examples/cookbook/dialogs.feature#L16) | |
|
|
244
|
-
| ✓ | `I rename the list to "Errands"` | DONE | [dialogs.feature:19](../examples/cookbook/dialogs.feature#L19) | The prompt gets the quoted value. With nothing quoted it is cancelled. |
|
|
245
|
-
| ✓ | `I leave by the "Search page" link` | DONE | [dialogs.feature:24](../examples/cookbook/dialogs.feature#L24) | The draft has unsaved text: "Leave site?" is accepted because the Step means to leave. Name the link: `I leave for the search page` split between it and the nav's `Search` link (51% / 49%), AMBIGUOUS in 2 of 3 runs. |
|
|
246
|
-
| ✗ | `I accept the confirm` | BLOCKED | [dialogs.feature:30](../examples/cookbook/dialogs.feature#L30) | The Step before it opened the confirm and already answered it (OK), so nothing is open. Say the intent in the Step that opens the dialog. |
|
|
247
|
-
|
|
248
|
-
## Attempt (negative paths)
|
|
249
|
-
|
|
250
|
-
| | Wording | Outcome | Example | Why |
|
|
251
|
-
|---|---|---|---|---|
|
|
252
|
-
| ✓ | `I try to schedule the campaign` | DONE | [campaigns.feature:21](../examples/campaign-manager/campaigns.feature#L21) | "try" is done once the action ran, even if the app refuses. The checks after it read the refusal. |
|
|
253
|
-
|
|
254
|
-
## Data tables and doc strings
|
|
255
|
-
|
|
256
|
-
An Action step's table runs as one Step per row: 2 columns are label/value pairs (`I enter "Ada" as First name`), 1
|
|
257
|
-
column a list (the Step text plus the value quoted), 3+ columns rows under a header. A check's table with 2+ columns
|
|
258
|
-
starts with its header and is matched row by row to the page table by column header (extra rows and any order pass;
|
|
259
|
-
"exactly" or "in this order" makes it exact); 1 column is a list of values that must all be shown ("should not show:" checks that none is). Code cannot check
|
|
260
|
-
that table rows are absent (UNVERIFIABLE). An action's table cells cannot be empty or hold a `"`: each becomes a quoted
|
|
261
|
-
value. A doc string is one more quoted value. Reports show the table compactly after the Step text.
|
|
262
|
-
|
|
263
|
-
| | Wording | Outcome | Example | Why |
|
|
264
|
-
|---|---|---|---|---|
|
|
265
|
-
| ✓ | `I fill in the new contact: [First name=Ada, Last name=Lovelace]` | DONE | [tables.feature:6](../examples/cookbook/tables.feature#L6) | Label/value pairs: one Step per row, each with its own Turns and effect check. |
|
|
266
|
-
| ✓ | `I check these roles: [Mathematician]` | DONE | [tables.feature:9](../examples/cookbook/tables.feature#L9) | A list: `I check these roles "Mathematician"`, so code confirms the box. |
|
|
267
|
-
| ✓ | `I write the notes: "Wrote the first program.\nBorn in London."` | DONE | [tables.feature:11](../examples/cookbook/tables.feature#L11) | The doc string is the value to type, line breaks kept. |
|
|
268
|
-
| ✓ | `the notes should say: "Wrote the first program.\nBorn in London."` | PASSED | [tables.feature:16](../examples/cookbook/tables.feature#L16) | Compared like a quoted value: spaces and line breaks count alike. |
|
|
269
|
-
| ✓ | `the contacts table should list: [First name=Ada, Roles=Mathematician; First name=Charles, Roles=Inventor]` | PASSED | [tables.feature:22](../examples/cookbook/tables.feature#L22) | Only the columns named in the header are compared; Jev names the table, code matches the rows. |
|
|
270
|
-
| ✓ | `the contacts table should show: [Lovelace, Babbage]` | PASSED | [tables.feature:26](../examples/cookbook/tables.feature#L26) | One column: every value must be shown in the table. |
|
|
271
|
-
| ✓ | `I add these contacts: [First name=Grace, Last name=Hopper, Roles=Inventor; First name=Alan, Last name=Turing, Roles=Mathematician]` | DONE | [tables.feature:31](../examples/cookbook/tables.feature#L31) | 3+ columns: a header, then one Step per row (`I add these contacts First name "Grace", ...`). |
|
|
272
|
-
| ✓ | `the contacts table should list exactly: [First name=Charles, Last name=Babbage, Roles=Inventor; First name=Grace, Last name=Hopper, Roles=Inventor; First name=Alan, Last name=Turing, Roles=Mathematician]` | PASSED | [tables.feature:35](../examples/cookbook/tables.feature#L35) | "exactly": the same rows, in this order, no others. |
|
|
273
|
-
| ✓ | `the contacts table should not show: [Lovelace, Knuth]` | PASSED | [tables.feature:40](../examples/cookbook/tables.feature#L40) | One column with "not": none of the values may be shown in the table. |
|
|
274
|
-
| ✗ | `the new contact should show: []` | UNVERIFIABLE | [tables.feature:48](../examples/cookbook/tables.feature#L48) | A check's first row is its header, so `\| First name \| Ada \|` alone compares nothing. Check the field instead (`the first name should be "Ada"`), or give the table a header row. |
|
|
275
|
-
| ✓ | `the contacts table should not show: [Babbage]` | ASSERTION_FAILED | [tables.feature:53](../examples/cookbook/tables.feature#L53) | Fails as it should: "Babbage" is in the table. Before the fix such a list was checked for presence and passed. |
|
|
276
|
-
|
|
277
|
-
## Value checks
|
|
278
|
-
|
|
279
|
-
| | Wording | Outcome | Example | Why |
|
|
280
|
-
|---|---|---|---|---|
|
|
281
|
-
| ✓ | `the campaign status should be "Draft"` | PASSED | [campaigns.feature:11](../examples/campaign-manager/campaigns.feature#L11) | One subject, one quoted value. |
|
|
282
|
-
| ✓ | `the error should say "Set an activation date before scheduling"` | PASSED | [campaigns.feature:22](../examples/campaign-manager/campaigns.feature#L22) | |
|
|
283
|
-
| ✓ | `the article heading should be "Ada Lovelace"` | PASSED | [lab.feature:7](../examples/cookbook/lab.feature#L7) | |
|
|
284
|
-
| ✓ | `the activation period should be "2 weeks"` | PASSED | [campaign-app.feature:12](../examples/campaign-app.feature#L12) | Reads the select's chosen option. |
|
|
285
|
-
| ✓ | `the campaign status should be "Scheduled"` | ASSERTION_FAILED | [campaign-app.feature:18](../examples/campaign-app.feature#L18) | Correct failure: a filter elsewhere on the page says "Scheduled", but the status says "Draft". Jev picks the subject by meaning. |
|
|
286
|
-
| ✗ | `the article title should be "Ada Lovelace"` | ASSERTION_FAILED | [lab.feature:13](../examples/cookbook/lab.feature#L13) | "Title" is also the browser tab's title ("Cookbook lab"). Say "heading". Jev reads it as the tab title and fails, or splits between the two (AMBIGUOUS): it fails either way. |
|
|
287
|
-
| ✗ | `the article title should be "Ada" or "Ada Lovelace"` | UNVERIFIABLE | [lab.feature:33](../examples/cookbook/lab.feature#L33) | A value check compares exactly one quoted value. |
|
|
288
|
-
| ✗ | `the article should be well written` | UNVERIFIABLE | [lab.feature:38](../examples/cookbook/lab.feature#L38) | Subjective: code cannot compare it. Quote something concrete, or opt in with `--judged`. |
|
|
289
|
-
|
|
290
|
-
## Presence and absence
|
|
291
|
-
|
|
292
|
-
| | Wording | Outcome | Example | Why |
|
|
293
|
-
|---|---|---|---|---|
|
|
294
|
-
| ✓ | `the article should say "born 10 December 1815"` | PASSED | [lab.feature:8](../examples/cookbook/lab.feature#L8) | Quoted text anywhere in the subject. |
|
|
295
|
-
| ✓ | `"Ada (programming language)" should appear in the results` | PASSED | [lab.feature:22](../examples/cookbook/lab.feature#L22) | Reported as a marginal pass: Jev was 61% on the "Results" region. Code still found the text there. |
|
|
296
|
-
| ✓ | `"Charles Babbage" should not appear in the results` | PASSED | [lab.feature:23](../examples/cookbook/lab.feature#L23) | |
|
|
297
|
-
| ✗ | `the results should be shown` | UNVERIFIABLE | [lab.feature:28](../examples/cookbook/lab.feature#L28) | Quotes nothing for code to look for. Quote an entry, as above. |
|
|
298
|
-
|
|
299
|
-
## Scoped presence (in an area)
|
|
300
|
-
|
|
301
|
-
| | Wording | Outcome | Example | Why |
|
|
302
|
-
|---|---|---|---|---|
|
|
303
|
-
| ✓ | `"Winter Sale" should appear in the calendar` | PASSED | [campaigns.feature:16](../examples/campaign-manager/campaigns.feature#L16) | Open the area first ([campaigns.feature:15](../examples/campaign-manager/campaigns.feature#L15)), then name it. |
|
|
304
|
-
| ✓ | `"Spring Sale" should not appear in the campaign list` | PASSED | [campaigns.feature:30](../examples/campaign-manager/campaigns.feature#L30) | |
|
|
305
|
-
| ✓ | `"Summer Sale" should appear in the calendar` | ASSERTION_FAILED | [scoped-presence.feature:9](../examples/probes/scoped-presence.feature#L9) | Correct failure: the draft was never scheduled. Its name in a toast elsewhere does not count. |
|
|
306
|
-
| ✗ | `"Spring Sale" should appear in the campaign list` | AMBIGUOUS | [scoped-presence.feature:13](../examples/probes/scoped-presence.feature#L13) | The list is not on this screen (the campaign's own page is), so the check cannot be decided. Open the list first. |
|
|
307
|
-
| ✗ | `the campaign list should contain "Spring Sale"` | AMBIGUOUS | [scoped-presence.feature:25](../examples/probes/scoped-presence.feature#L25) | Same: the named area is not on the screen. |
|
|
308
|
-
|
|
309
|
-
## State checks
|
|
310
|
-
|
|
311
|
-
| | Wording | Outcome | Example | Why |
|
|
312
|
-
|---|---|---|---|---|
|
|
313
|
-
| ✓ | `the "Schedule" button should be disabled` | PASSED | [campaigns.feature:14](../examples/campaign-manager/campaigns.feature#L14) | Quote the control's name. Name the state: disabled, enabled, checked, unchecked, expanded, collapsed, selected, pressed. |
|
|
314
|
-
| ✓ | `the schedule button should be enabled` | PASSED | [memory.feature:24](../examples/probes/memory.feature#L24) | Unquoted name also works when it is unique. |
|
|
315
|
-
| ✓ | `"Newsletter" should be unchecked` | PASSED | [lab.feature:43](../examples/cookbook/lab.feature#L43) | |
|
|
316
|
-
|
|
317
|
-
## Memory (a check that refers back)
|
|
318
|
-
|
|
319
|
-
| | Wording | Outcome | Example | Why |
|
|
320
|
-
|---|---|---|---|---|
|
|
321
|
-
| ✓ | `the campaign should appear in the calendar` | PASSED | [memory.feature:11](../examples/probes/memory.feature#L11) | "the campaign" is the one an earlier Step quoted ("Winter Sale"). |
|
|
322
|
-
| ✗ | `the campaign should appear in the calendar` | UNVERIFIABLE | [memory.feature:19](../examples/probes/memory.feature#L19) | Same wording, but Jev was only 68% sure which earlier value it meant. Below the gate it refuses rather than guess. Quote the value when the scenario names several. |
|
|
323
|
-
|
|
324
|
-
## Secrets
|
|
325
|
-
|
|
326
|
-
| | Wording | Outcome | Example | Why |
|
|
327
|
-
|---|---|---|---|---|
|
|
328
|
-
| ✓ | `I log in as "$SAUCE_USER" with password "$SAUCE_PASSWORD"` | DONE | [secret.feature:6](../examples/probes/secret.feature#L6) | `"$NAME"` comes from the environment. Jev, logs and cassette see only the token. |
|
|
329
|
-
| ✓ | `the page heading should say "Products"` | PASSED | [secret.feature:7](../examples/probes/secret.feature#L7) | |
|
|
330
|
-
|
|
331
|
-
## Clear fields, native dates and on-leave validation
|
|
332
|
-
|
|
333
|
-
Clear/empty offers an empty TYPE Candidate value. Quote ISO values for native date (`YYYY-MM-DD`), time (`HH:mm`)
|
|
334
|
-
and datetime-local (`YYYY-MM-DDTHH:mm`) inputs; custom calendars remain multi-Turn clicks. TYPE blurs to trigger
|
|
335
|
-
on-leave validation, then restores focus for Enter only if the application left focus on the body and the field
|
|
336
|
-
remains connected, visible and enabled. Application-directed focus and blur dialogs are not overridden.
|
|
337
|
-
|
|
338
|
-
| | Wording | Outcome | Example | Why |
|
|
339
|
-
|---|---|---|---|---|
|
|
340
|
-
| ✓ | `I clear the Search field` | DONE | [form-details.feature:4](../examples/cookbook/form-details.feature#L4) | TYPE with an empty value. |
|
|
341
|
-
| ✓ | `the search status should be "Search cleared"` | PASSED | [form-details.feature:5](../examples/cookbook/form-details.feature#L5) | Validation runs after clearing. |
|
|
342
|
-
| ✓ | `I set the Start date to "2026-10-20"` | DONE | [form-details.feature:8](../examples/cookbook/form-details.feature#L8) | Native ISO date. |
|
|
343
|
-
| ✓ | `I set the Start time to "14:30"` | DONE | [form-details.feature:10](../examples/cookbook/form-details.feature#L10) | Native time. |
|
|
344
|
-
| ✓ | `I set the Appointment to "2026-10-20T14:30"` | DONE | [form-details.feature:12](../examples/cookbook/form-details.feature#L12) | Native local date/time. |
|
|
345
|
-
| ✓ | `I enter "invalid" in Email` | DONE | [form-details.feature:16](../examples/cookbook/form-details.feature#L16) | Entering text can succeed while validation rejects it. |
|
|
346
|
-
| ✓ | `the email status should be "Invalid email"` | PASSED | [form-details.feature:17](../examples/cookbook/form-details.feature#L17) | Checks the on-leave error. |
|