@asrunjs/asrun 0.5.2 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +10 -0
- package/LICENSE +9 -21
- package/README.md +178 -469
- package/dist/scripts/go.mjs +188 -188
- package/dist/scripts/init.js +1 -1
- package/docs/agents/case-format.md +193 -0
- package/docs/agents/framework-dev.md +94 -6
- package/docs/ai-workflow.md +235 -0
- package/docs/cases.md +189 -0
- package/docs/commands.md +175 -0
- package/docs/install.md +174 -0
- package/docs/profiles.md +139 -0
- package/docs/troubleshooting.md +101 -0
- package/docs/web-ui.md +139 -0
- package/package.json +10 -2
- package/templates/AGENTS.md +2 -0
- package/ui/dist/assets/{dist-DSy3dVf4.js → dist-BudNAjOv.js} +1 -1
- package/ui/dist/assets/{dist-Ba-HC662.js → dist-C-eCSFja.js} +1 -1
- package/ui/dist/assets/{dist-C-Oyo1ZT.js → dist-CO4heSzh.js} +1 -1
- package/ui/dist/assets/{dist-C8MdhqgG.js → dist-CPLhRzom.js} +1 -1
- package/ui/dist/assets/{dist-Dq6sW5Ur.js → dist-DEPJZ3Me.js} +1 -1
- package/ui/dist/assets/{dist-CmpteMYY.js → dist-DZ7RiqmA.js} +1 -1
- package/ui/dist/assets/{dist-BGbe0-Aa.js → dist-LixzGniF.js} +1 -1
- package/ui/dist/assets/{dist-CG1pHn5m.js → dist-O_RVi4tP.js} +1 -1
- package/ui/dist/assets/{dist-BtMLMthJ.js → dist-jPS6HCJ5.js} +1 -1
- package/ui/dist/assets/{dist-jjxQutEm.js → dist-vmf3fnkM.js} +1 -1
- package/ui/dist/assets/{index-P_Yo0ZZn.js → index-BIGt8Hfz.js} +3 -3
- package/ui/dist/index.html +1 -1
|
@@ -4,26 +4,50 @@ Read `AGENTS.md` first: it is the product baseline, and this file only adds how
|
|
|
4
4
|
is relevant to a project that uses asrun as a dependency.
|
|
5
5
|
|
|
6
6
|
## Framework layout
|
|
7
|
+
|
|
7
8
|
`<framework>` for a project is `node_modules/@asrunjs/asrun`. In this repository it is the root folder:
|
|
8
|
-
|
|
9
|
+
|
|
10
|
+
- `lib/` fixtures (`profile`, session guard, write gate) and helpers; specs import them as
|
|
9
11
|
`@asrunjs/asrun/<module>` (for example `@asrunjs/asrun/fixtures`, `@asrunjs/asrun/values`)
|
|
10
|
-
- `scripts/` user-facing commands
|
|
12
|
+
- `scripts/` the user-facing commands; every command and flag is documented in `docs/commands.md`
|
|
11
13
|
- `templates/` the files `pnpm asrun init` writes into a workspace
|
|
12
14
|
- `ui/` local web UI (`pnpm ui`). It may only run the fixed actions in `ui/server/actions.ts`
|
|
13
15
|
and must never serve anything under `.auth/`.
|
|
14
|
-
- `docs
|
|
16
|
+
- `docs/` user documentation (`install.md`, `profiles.md`, `cases.md`, `ai-workflow.md`,
|
|
17
|
+
`commands.md`, `web-ui.md`, `troubleshooting.md`) and `docs/theme-guide.html`
|
|
18
|
+
- `docs/agents/` documents for AI agents: this file, and `case-format.md` (the case document contract)
|
|
15
19
|
- `examples/demo/` a workspace for developing the framework itself against a local Docker stack (Conduit);
|
|
16
20
|
it is committed (`.auth/` and `reports/` never are), see its README
|
|
21
|
+
- `.claude/` skills + slash commands for Claude Code; the skills are shared with OpenCode
|
|
22
|
+
- `.opencode/commands/` the same commands for OpenCode (it does not read `.claude/commands`)
|
|
23
|
+
- `AGENTS.md` the product baseline rules (OpenCode reads it); `CLAUDE.md` imports it for Claude Code
|
|
24
|
+
- `README.md` the user-facing overview: what it does, install, first test, cookbook, links to `docs/`
|
|
25
|
+
|
|
26
|
+
`package.json` `files` ships `README.md`, `AGENTS.md`, `CLAUDE.md`, `docs/`, `templates/`, `.claude/`, `bin/`,
|
|
27
|
+
`dist/` and `ui/dist/`, so a project can read the same documentation from
|
|
28
|
+
`node_modules/@asrunjs/asrun/docs/`.
|
|
17
29
|
|
|
18
30
|
## Working on the framework itself
|
|
31
|
+
|
|
19
32
|
- The framework repository holds no workspace content at the root: no `profiles.yaml`, `cases/`, `tests/` or
|
|
20
33
|
reports there. The one workspace is `examples/demo`, which is committed because it runs its own local site; its
|
|
21
|
-
`.auth/`, `reports/` and other run output are not.
|
|
22
|
-
|
|
23
|
-
|
|
34
|
+
`.auth/`, `reports/` and other run output are not.
|
|
35
|
+
- Develop against the demo:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
pnpm demo:up # build and start the local stack
|
|
39
|
+
pnpm demo:seed # accounts, articles, follows, favorites
|
|
40
|
+
pnpm asrun init --root examples/demo # also writes the framework's .env (ASRUN_ROOT=examples/demo)
|
|
41
|
+
pnpm build # the demo's specs import the package, which resolves to dist/
|
|
42
|
+
pnpm asrun --env staging --read-only
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
`pnpm demo:reset` deletes the database and seeds it again; logins must be redone afterwards. Demo accounts are in
|
|
46
|
+
`examples/demo/README.md`.
|
|
24
47
|
- Everything the framework reads for a project goes through `ROOT` in `lib/paths.ts` (the workspace root); its own
|
|
25
48
|
assets (templates, docs, skills, its `node_modules`) go through `FRAMEWORK_ROOT`. Never write into
|
|
26
49
|
`FRAMEWORK_ROOT` at run time.
|
|
50
|
+
- After changing `lib/`, run `pnpm build` before exercising the demo's specs.
|
|
27
51
|
- `templates/` is the source of the files a workspace gets. When a template changes, bump `version` in
|
|
28
52
|
`package.json` so `pnpm asrun doctor` in a project shows that it is behind.
|
|
29
53
|
- `init` owns two lists: `GENERATED` (rewritten by `--update`; the commands and the Playwright files) and `ONCE`
|
|
@@ -33,7 +57,51 @@ is relevant to a project that uses asrun as a dependency.
|
|
|
33
57
|
- A project's rules never flow back here. The framework reads a workspace only through `ROOT` at run time; nothing
|
|
34
58
|
under `templates/` or `.claude/skills/` is generated from a project.
|
|
35
59
|
|
|
60
|
+
### Installing this repository as a Git dependency (private)
|
|
61
|
+
|
|
62
|
+
The npm package is the public channel for users. This repository is private, so a Git dependency works only on a
|
|
63
|
+
machine that can clone it:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
pnpm add -D github:wizcas/asrun#<tag>
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
pnpm 12 resolves a GitHub spec through the canonical HTTPS URL, not SSH. That machine therefore needs HTTPS
|
|
70
|
+
credentials (a credential helper or a token), or a rewrite to SSH:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
git config --global url."git@github.com:".insteadOf https://github.com/
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
On CI, give the runner a `GITHUB_TOKEN`, for example:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
git config --global \
|
|
80
|
+
url."https://x-access-token:${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
or a deploy key with the SSH rewrite above. A Git install also runs the package's `prepare` (it builds `dist/` and
|
|
84
|
+
`ui/dist/`), so it is slower than the registry: use it only to pin a commit, a branch or a tag that is not published
|
|
85
|
+
yet. For a one-off change, `pnpm patch @asrunjs/asrun` and commit the patch instead.
|
|
86
|
+
|
|
87
|
+
## Documentation
|
|
88
|
+
|
|
89
|
+
The split is intentional; keep it:
|
|
90
|
+
|
|
91
|
+
- `README.md` — the user's first page: what AsRun does, requirements, install, the first test, customizing the
|
|
92
|
+
prompts, the every-day cookbook, safety in short, a documentation index, maintainer notes. It should stay readable
|
|
93
|
+
in one sitting; a section that grows past a screen belongs in `docs/`.
|
|
94
|
+
- `docs/*.md` — the user reference. `docs/commands.md` is the only place that lists every command and flag.
|
|
95
|
+
- `docs/agents/*.md` — for AI agents. `case-format.md` is the contract of the case document (fixed tokens, values,
|
|
96
|
+
targets); `AGENTS.md` references it, and the template `AGENTS.md` does too.
|
|
97
|
+
|
|
98
|
+
When behavior changes, update the matching document in the same change. A command or flag goes into
|
|
99
|
+
`docs/commands.md` and, if it is a daily one, into the README cookbook. A change to the case format goes into
|
|
100
|
+
`docs/agents/case-format.md` and `docs/cases.md` together. Keep the relative links working and the fixed tokens
|
|
101
|
+
spelled exactly as the parser reads them.
|
|
102
|
+
|
|
36
103
|
## The commands and the two agent CLIs
|
|
104
|
+
|
|
37
105
|
- Claude Code reads `.claude/commands/*` and `.claude/skills/<name>/SKILL.md`.
|
|
38
106
|
- OpenCode reads `.opencode/commands/*`, `.claude/skills/<name>/SKILL.md`, `.opencode/skills/<name>/SKILL.md` and
|
|
39
107
|
`.agents/skills/<name>/SKILL.md`. It does **not** read `.claude/commands`, and it does **not** read a flat
|
|
@@ -46,8 +114,10 @@ is relevant to a project that uses asrun as a dependency.
|
|
|
46
114
|
plain file read. That is why the skill itself must read the project override (`.claude/skills/<skill>.md`).
|
|
47
115
|
|
|
48
116
|
## Web UI (`ui/web`)
|
|
117
|
+
|
|
49
118
|
Before adding or changing any UI element, read `docs/theme-guide.html` (open the file; it is static HTML) and follow it.
|
|
50
119
|
Its rules are binding, in particular:
|
|
120
|
+
|
|
51
121
|
- Icons: Feather via the `react-feather` package only (`import {ChevronDown} from 'react-feather'`). Never draw, paste
|
|
52
122
|
or hand-write SVGs, and never add another icon library. Size 14 in buttons, 16-20 elsewhere; decorative icons get
|
|
53
123
|
`aria-hidden="true"`, icon-only buttons get an `aria-label`. Do not use Unicode symbols (`▲ ▼ ◀ ▶ ⤢`) as icons.
|
|
@@ -59,6 +129,9 @@ Its rules are binding, in particular:
|
|
|
59
129
|
`ui/web/src/components/search-select.tsx`: an input with a filtered list below it, which a native `<select>` cannot
|
|
60
130
|
do. It keeps `Select`'s height, border and focus ring, so the two look like one family; do not build a third variant.
|
|
61
131
|
- If a change adds or alters a token, update `ui/web/src/index.css` and `docs/theme-guide.html` together.
|
|
132
|
+
- The Help page ("How to start") shows screenshots from `ui/web/public/help/`: retake one when the page it shows
|
|
133
|
+
changes.
|
|
134
|
+
|
|
62
135
|
Before reaching for a new dependency or writing a helper component, grep `package.json` and `ui/web/src` for what the
|
|
63
136
|
project already uses.
|
|
64
137
|
|
|
@@ -77,3 +150,18 @@ Every lead, label and tooltip is written for the person using the UI, never for
|
|
|
77
150
|
`CopyBlock`, or a folded `<details>` block titled "For the command line".
|
|
78
151
|
- **Say what the user sees.** "The run appears in the Jobs bar", not "the command runs in a one-shot session"; "the
|
|
79
152
|
browser shows a small card", not "the agent pauses on a `writes` step".
|
|
153
|
+
|
|
154
|
+
## Before you open a pull request
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
pnpm typecheck
|
|
158
|
+
pnpm build # dist/ must match lib/ and scripts/
|
|
159
|
+
pnpm case:check # examples/demo cases
|
|
160
|
+
pnpm spec:lint # examples/demo specs
|
|
161
|
+
pnpm ui:build # ui/dist/ must match ui/web and ui/server
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
- `pnpm build` and `pnpm ui:build` also run from `prepare` and `prepack`, so a published package is complete.
|
|
165
|
+
- Bump `version` when `templates/` or the shipped docs change; `pnpm asrun doctor` in a project compares it.
|
|
166
|
+
- Keep the demo's committed expectations (`profiles.yaml`, `cases/`, `tests/`, baselines) in step with the demo site.
|
|
167
|
+
Baselines are a person's decision: never let an agent regenerate them.
|
|
@@ -0,0 +1,235 @@
|
|
|
1
|
+
# The AI workflow
|
|
2
|
+
|
|
3
|
+
Four commands carry a test case from rough notes to a Playwright spec, and repair it when the product changes. They
|
|
4
|
+
work the same in Claude Code and OpenCode: each command is a thin shell that loads a skill from the framework package,
|
|
5
|
+
so the procedure lives in one place and a project can refine it (see
|
|
6
|
+
[Customizing the prompts](#customizing-the-prompts)).
|
|
7
|
+
|
|
8
|
+
| Command | What it does |
|
|
9
|
+
|---|---|
|
|
10
|
+
| `/make-case <notes-or-path>` | Turn raw notes into case documents. Optional step. |
|
|
11
|
+
| `/explore-case <case-id> <env> [--attended]` | Open the real pages and write what they show into the case. |
|
|
12
|
+
| `/gen-spec <case-id> <env> [--force]` | Write `tests/<case-id>.spec.ts` and fill the case's `## Values`. |
|
|
13
|
+
| `/heal-spec <case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]` | Diagnose a failing spec from its last report and fix it. |
|
|
14
|
+
|
|
15
|
+
The environment is **required** by every command except `/make-case`, which does not open pages. There is no default.
|
|
16
|
+
|
|
17
|
+
Every command ends with `RESULT: ready` or `RESULT: needs-input`. `needs-input` means somebody has to answer a
|
|
18
|
+
question in the case file (or, in the web UI, in the job's **Answer** sheet). A one-shot run cannot ask in the chat:
|
|
19
|
+
it writes its questions into the case and stops.
|
|
20
|
+
|
|
21
|
+
In the web UI you never type a command: choose the step in a case toolbar and press **Run AI**, or paste notes on
|
|
22
|
+
**Cases > From memo**. The UI runs the selected AI tool and streams the output into the Jobs bar.
|
|
23
|
+
|
|
24
|
+
## 0. Optional: start from raw notes
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
/make-case path/to/notes.md
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
or paste the text after the command. The agent normalizes the notes into steps in the project's artifact language
|
|
31
|
+
(English by default), marks the steps that write data (`[writes]`) or send email (`[email]`), keeps the original lines
|
|
32
|
+
in a `## Source` table, and lists the gaps with a suggested default for each. It never invents expected values
|
|
33
|
+
(unknown ones become a sentence with a `TODO` block under it) and never opens a browser. What the pages really show
|
|
34
|
+
is found in step 2.
|
|
35
|
+
|
|
36
|
+
It writes `cases/<id>.md` only, and never overwrites an existing case. `pnpm case:check` validates the result.
|
|
37
|
+
|
|
38
|
+
## 1. Write or open the case
|
|
39
|
+
|
|
40
|
+
`pnpm case:new header-logo` creates `cases/header-logo.md` from the template, or use a case you already have. The
|
|
41
|
+
format is in [cases.md](./cases.md). Write the steps, the expectations and — when you know them — the values.
|
|
42
|
+
|
|
43
|
+
## 2. Explore the case (only where a person and the agent work together)
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
/explore-case header-logo staging
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The agent opens the real pages of the profiles that are logged in and only **reads** them. It never creates, edits,
|
|
50
|
+
publishes, shares or deletes anything, in staging or in production. It:
|
|
51
|
+
|
|
52
|
+
- fixes wording in the case file and writes what the pages show into it, marked `observed, 🔍 to be confirmed by QA`;
|
|
53
|
+
- turns what it cannot know into `🚫 TODO [blocking]` / `❓ TODO [non-blocking]` questions, each under a sentence that
|
|
54
|
+
is its suggestion and can stand in the case as it is, with a last line `- ✍️ Answer:`;
|
|
55
|
+
- lists the candidates instead of guessing a target (`## Targets`);
|
|
56
|
+
- edits the case file only (plus reference screenshots under `.explore/targets/`).
|
|
57
|
+
|
|
58
|
+
**To accept the suggestion, delete the whole block.** **To answer, write after `Answer:`.** Then run the command
|
|
59
|
+
again. In the web UI, the run's **What to do next** bar offers **Answer**: the question list, an answer box,
|
|
60
|
+
**Use the suggested defaults** and **Generate spec**. A continued job keeps the same conversation and permissions, and
|
|
61
|
+
can be continued once per turn, only while the UI server has not restarted.
|
|
62
|
+
|
|
63
|
+
### Attended exploration (`--attended`)
|
|
64
|
+
|
|
65
|
+
Use it for a case that creates or edits data, or needs a choice (an existing document, items). Stay at the computer:
|
|
66
|
+
the browser window opens and the agent waits while you do those steps.
|
|
67
|
+
|
|
68
|
+
- The window shows a small draggable card in the bottom-left corner with what to do, **Done** (hand control back) and
|
|
69
|
+
**Cannot do this**. The agent also continues by itself when the finish condition it named shows up (for example a
|
|
70
|
+
"Saved" message); the card turns green when control is back. It waits up to 10 minutes per step.
|
|
71
|
+
- What you did is recorded in the case under the step as **general recipe lines**
|
|
72
|
+
(`- Recipe (phase): ...`): positions instead of names of changeable data, `[asrun]` names for created data, control
|
|
73
|
+
labels but no selectors.
|
|
74
|
+
- Delete the `[asrun]` items you created before you press Done on the last step.
|
|
75
|
+
- Without `--attended` the agent never waits: a step that needs a person becomes a `TODO [blocking]`.
|
|
76
|
+
|
|
77
|
+
In the web UI the **Attended** box is ticked by default for Explore.
|
|
78
|
+
|
|
79
|
+
## 3. Log in the profiles the case needs
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
pnpm status
|
|
83
|
+
pnpm signin example-staging
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Skip this for a `login: none` case. Sessions are per profile; see [profiles.md](./profiles.md).
|
|
87
|
+
|
|
88
|
+
## 4. Generate the spec
|
|
89
|
+
|
|
90
|
+
```
|
|
91
|
+
/gen-spec header-logo staging
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
The agent replays the case on the real pages with your saved sessions (it never writes data) and writes
|
|
95
|
+
`tests/header-logo.spec.ts`. It fills the case's `## Values` for the profiles it explored and checks the files
|
|
96
|
+
(`pnpm typecheck`, `pnpm case:check`; `pnpm spec:lint <case-id>` must have no failure).
|
|
97
|
+
|
|
98
|
+
It does **not** run the spec and does **not** create screenshot baselines: running is a separate step (6).
|
|
99
|
+
|
|
100
|
+
- A case must be in the project's artifact language. If it is in another language, `/gen-spec` translates the case
|
|
101
|
+
file first and tells you.
|
|
102
|
+
- A `[writes]` or `[input]` step with no `- Recipe:` line under it is not generated: the command stops with
|
|
103
|
+
`needs-input` and asks you to run step 2 with `--attended`.
|
|
104
|
+
- An existing spec is left alone; `--force` regenerates it from the case. The case's `## Values` and the baseline
|
|
105
|
+
images are kept, and the report lists the baselines that may be out of date. In the web UI, **Run** on a case that
|
|
106
|
+
has a spec asks "Overwrite the spec?" first.
|
|
107
|
+
|
|
108
|
+
## 5. Review what the AI produced
|
|
109
|
+
|
|
110
|
+
Check that the spec reads like the case, and that `## Values` has the right values for each profile. The spec imports
|
|
111
|
+
`test`/`expect` from `@asrunjs/asrun/fixtures`, never from `@playwright/test`, and every step is a `test.step` named
|
|
112
|
+
after the case's step. You normally do not edit either file by hand, but you can.
|
|
113
|
+
|
|
114
|
+
## 6. Run it and read the evidence
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
pnpm asrun --env staging --grep header-logo
|
|
118
|
+
pnpm report
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
The first run of a spec with visual checks writes its baselines. Run it once with `--update-snapshots`, then open
|
|
122
|
+
every new image under `tests/<case>.spec.ts-snapshots/` and confirm it shows the target card's element for that
|
|
123
|
+
profile. Baselines are the source of truth for pixel checks.
|
|
124
|
+
|
|
125
|
+
Then run all profiles and read the report: the video of each test, screenshots on failure, and the trace with the page
|
|
126
|
+
and the network requests. See [commands.md](./commands.md) for filters and baselines.
|
|
127
|
+
|
|
128
|
+
## Healing a failing spec
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
/heal-spec header-logo staging
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
or **Heal spec** in the case toolbar, or **Heal this failure** on a failed result of the Reports page.
|
|
135
|
+
|
|
136
|
+
The agent decides whether the test needs updating or whether it found a real product problem. It **never** edits an
|
|
137
|
+
expectation just to make a failure disappear.
|
|
138
|
+
|
|
139
|
+
- The environment is required. Only the profiles that are logged in there run (`--logged-in` leaves the others out
|
|
140
|
+
instead of failing them).
|
|
141
|
+
- The web UI first opens a dialog: which profiles to heal (every logged-in profile ticked) and an optional text box
|
|
142
|
+
for what you want healed, with evidence or clues. The text is stored in `.tmp/hints/` and passed as `--hints`.
|
|
143
|
+
Clues guide the diagnosis; they never lower the bar for a product finding.
|
|
144
|
+
- It reads the existing report first:
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
pnpm heal:evidence --case header-logo --env staging
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
It prints status, failed step, error, call log and screenshots per profile, and whether the report is fresh or
|
|
151
|
+
stale. The case is run again only when there is no report, the report is stale, it does not explain the failure, or
|
|
152
|
+
a clue has to be checked. After editing the spec, it runs the case once to confirm.
|
|
153
|
+
|
|
154
|
+
## Why a generated spec can be trusted
|
|
155
|
+
|
|
156
|
+
Three binding rules hold in exploring, generating and healing. They live in the framework's `AGENTS.md` and in
|
|
157
|
+
`.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`.
|
|
158
|
+
|
|
159
|
+
- **Targets.** A check on a logo, image, icon, file, PDF, screenshot or repeated text points to a `[T1]` card under
|
|
160
|
+
`## Targets`: what it is (and what it is not), where it is, how it is identified (never `.first()` or a bare `img`),
|
|
161
|
+
when it is ready, and a highlighted reference screenshot under `.explore/targets/`. When in doubt the agent does not
|
|
162
|
+
pick: it writes a `🚫 TODO [blocking]` that names the exact image, text, place or UI, with the candidates listed.
|
|
163
|
+
- **Readiness.** After an action that loads something, the key requests must finish and the content must be visible
|
|
164
|
+
before anything is read. Specs use `actAndWait` (`@asrunjs/asrun/ready`); "missing" and "empty" checks need a
|
|
165
|
+
positive ready signal first. Healing treats a "missing / empty / timeout" failure as a readiness problem first.
|
|
166
|
+
- **Downloads.** `verifyDownload` (`@asrunjs/asrun/downloads`) checks that the download starts, finishes, the file
|
|
167
|
+
exists and the file is valid for its kind. Any failure or timeout fails the whole step with a message that names the
|
|
168
|
+
stage.
|
|
169
|
+
- **PDFs are read like a person reads them.** `expectPdf` (`@asrunjs/asrun/pdf`) renders the pages to images (a logo
|
|
170
|
+
region must not be blank and must match the profile's baseline, 2% tolerance) and extracts the text. Every page is
|
|
171
|
+
checked unless the case limits the pages in strong words ("only", "exactly", "must"); "should", "always", "usually"
|
|
172
|
+
or no wording mean every page.
|
|
173
|
+
|
|
174
|
+
Checks you can run yourself:
|
|
175
|
+
|
|
176
|
+
| Command | What it checks |
|
|
177
|
+
|---|---|
|
|
178
|
+
| `pnpm spec:lint [case-id]` | fails on screenshots without a target, `.first()` on images, sleeps, `networkidle`, raw download events, a `%PDF` check on its own |
|
|
179
|
+
| `pnpm baselines:review [case-id] [--new]` | lists every baseline with the card that says what it must show, so you can look at each new image |
|
|
180
|
+
| `pnpm case:check [case-id]` | fails on an incomplete card or bad values; warns on an image check with no card |
|
|
181
|
+
| `pnpm pdf:render <file.pdf> --out <dir> [--pages all] [--corner top-left]` | renders the pages to images and prints the text, to look at a PDF by hand |
|
|
182
|
+
|
|
183
|
+
## Customizing the prompts
|
|
184
|
+
|
|
185
|
+
The commands are shells; the procedure is a skill in the framework package:
|
|
186
|
+
|
|
187
|
+
- commands: `.claude/commands/*.md` (Claude Code) and `.opencode/commands/*.md` (OpenCode);
|
|
188
|
+
- skills: `<framework>/.claude/skills/{making-cases,exploring-cases,generating-specs}/`.
|
|
189
|
+
|
|
190
|
+
A project refines a skill with a **flat override file**, named after the skill (not the command):
|
|
191
|
+
|
|
192
|
+
```
|
|
193
|
+
.claude/skills/generating-specs.md
|
|
194
|
+
.claude/skills/exploring-cases.md
|
|
195
|
+
.claude/skills/making-cases.md
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
The command and the skill read it; it is not discovered as a standalone skill. Rules combine in this order, and a
|
|
199
|
+
later one wins where they conflict:
|
|
200
|
+
|
|
201
|
+
1. `F1`–`F6` of the framework baseline: safety, in every project. Nothing can change them.
|
|
202
|
+
2. The framework baseline, `<framework>/AGENTS.md`.
|
|
203
|
+
3. The framework skill.
|
|
204
|
+
4. The project override `.claude/skills/<skill>.md`.
|
|
205
|
+
5. The project's own `AGENTS.md` §2 (Project rules).
|
|
206
|
+
|
|
207
|
+
An override is plain Markdown; write only what differs:
|
|
208
|
+
|
|
209
|
+
```markdown
|
|
210
|
+
# generating-specs in this project
|
|
211
|
+
|
|
212
|
+
- Step titles in `test.step` keep the case's wording, including the number.
|
|
213
|
+
- A helper used by more than one spec goes in `tests/helpers/<name>.ts`.
|
|
214
|
+
- The staging account has no published documents: assert the empty state there instead.
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
The project's `AGENTS.md` §2 is the other place: a table of rule ids (`F7`–`F10`; `F1`–`F6` are fixed) with `narrows`
|
|
218
|
+
(stricter) or `replaces` (a different default). For example, a project that writes cases in Simplified Chinese
|
|
219
|
+
`replaces` `F10`. `pnpm asrun doctor` fails on an unknown id, a missing `Effect`, or a non-negotiable rule, and lists
|
|
220
|
+
the overrides it sees.
|
|
221
|
+
|
|
222
|
+
An update never overwrites `AGENTS.md`, `CLAUDE.md` or `.claude/skills/**`: `pnpm asrun init --force` refuses them
|
|
223
|
+
without `--reset-rules`. Never edit the framework package itself.
|
|
224
|
+
|
|
225
|
+
Project-level AI tool configuration:
|
|
226
|
+
|
|
227
|
+
| File | What it sets |
|
|
228
|
+
|---|---|
|
|
229
|
+
| `.mcp.json` | the `playwright-test` MCP server for Claude Code |
|
|
230
|
+
| `opencode.jsonc` | the same MCP server for OpenCode, plus an optional `"model"` for one-shot runs |
|
|
231
|
+
| `.claude/settings.json` | the sandbox exceptions for `pnpm asrun`, `pnpm status`, `pnpm heal:evidence` |
|
|
232
|
+
|
|
233
|
+
In the web UI, the top bar chooses the AI tool and the model for every AI run, and **Thinking** passes `--thinking`
|
|
234
|
+
to OpenCode. The choice is remembered per CLI in the browser. Without a model, a one-shot OpenCode run uses its
|
|
235
|
+
built-in free model, which has a daily usage limit.
|
package/docs/cases.md
ADDED
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
# Writing case documents
|
|
2
|
+
|
|
3
|
+
A case is a Markdown file that a person writes and reviews. It is the source of truth for the test: steps,
|
|
4
|
+
expectations, the exact values and the visual targets. The spec is generated from it and never replaces it.
|
|
5
|
+
|
|
6
|
+
## Where cases live
|
|
7
|
+
|
|
8
|
+
- `cases/<name>.md`, or one group folder: `cases/<group>/<name>.md`.
|
|
9
|
+
- The **id** is the path below `cases/` without `.md`: `header-logo` or `documents/landing-page`. Names are lowercase
|
|
10
|
+
words with single dashes.
|
|
11
|
+
- The spec of a case lives at the same path below `tests/` (`tests/documents/landing-page.spec.ts`), with its
|
|
12
|
+
`*-snapshots/` folder next to it.
|
|
13
|
+
- Files starting with `_` are ignored (`cases/_template.md` is the template).
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
pnpm case:new header-logo # creates cases/header-logo.md from the template
|
|
17
|
+
pnpm case:new documents/header-logo # also creates the group folder
|
|
18
|
+
pnpm case:check # validates every case (or: pnpm case:check <id>)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## The document
|
|
22
|
+
|
|
23
|
+
````markdown
|
|
24
|
+
---
|
|
25
|
+
id: header-logo
|
|
26
|
+
title: The header shows the product logo
|
|
27
|
+
login: required
|
|
28
|
+
suites: [smoke, regression]
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## Goal
|
|
32
|
+
Why this case exists, in one or two sentences.
|
|
33
|
+
|
|
34
|
+
## Steps
|
|
35
|
+
1. Open `/`.
|
|
36
|
+
- Ready when: the header has rendered its menu.
|
|
37
|
+
2. The header shows the product logo. [T1]
|
|
38
|
+
|
|
39
|
+
## Expected
|
|
40
|
+
- The logo is the product logo for this profile, not a customer or partner logo.
|
|
41
|
+
|
|
42
|
+
## Values
|
|
43
|
+
```yaml
|
|
44
|
+
default:
|
|
45
|
+
logoAlt: Acme
|
|
46
|
+
example-staging:
|
|
47
|
+
logoAlt: Acme Staging
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Visual checks
|
|
51
|
+
- The header logo matches its baseline. [T1]
|
|
52
|
+
|
|
53
|
+
## Targets
|
|
54
|
+
### T1 Header logo
|
|
55
|
+
- What: the product logo, not the customer logo in the banner.
|
|
56
|
+
- Where: top-left of the header.
|
|
57
|
+
- Identified by: role `img`, name "Acme".
|
|
58
|
+
- Not: the customer logo in the banner and the favicon.
|
|
59
|
+
- Ready when: the image has a non-zero natural width.
|
|
60
|
+
- Reference example-staging: .explore/targets/header-logo/T1-example-staging.png
|
|
61
|
+
|
|
62
|
+
## Notes
|
|
63
|
+
Anything a reviewer should know.
|
|
64
|
+
````
|
|
65
|
+
|
|
66
|
+
The headings are fixed English tokens (`Goal`, `Steps`, `Expected`, `Values`, `Visual checks`, `Targets`, `Notes`,
|
|
67
|
+
and `Source` when the case was made from raw notes). A project language never translates them.
|
|
68
|
+
|
|
69
|
+
## Front matter
|
|
70
|
+
|
|
71
|
+
| Key | Meaning |
|
|
72
|
+
|---|---|
|
|
73
|
+
| `id` | The case id (the path without `.md`). |
|
|
74
|
+
| `title` | One-line summary, shown in the editor and the spec. |
|
|
75
|
+
| `login` | `required` (the default) or `none` for a public page that needs no saved session. A `login: none` spec sets `test.use({mustLogin: false})`. |
|
|
76
|
+
| `suites` | Required. One or more lowercase words: `suites: [smoke, regression]`. A case runs in every suite it lists. |
|
|
77
|
+
| `writes` | The environments in which the case may create or change data, for example `writes: [staging]`. Omit it for a read-only case. |
|
|
78
|
+
| `writes-note` | Required with `writes`: what the case creates or changes, and that it is removed again. |
|
|
79
|
+
| `envs` | Optional: run the case only in these environments; elsewhere it is skipped. Must include every environment in `writes`. |
|
|
80
|
+
|
|
81
|
+
`priority: <level>` is the old name of `suites`; it is read as `suites: [<level>]`. Do not use both.
|
|
82
|
+
|
|
83
|
+
## Steps
|
|
84
|
+
|
|
85
|
+
Numbered lines. Keep them short and in plain language: the spec uses each step's text verbatim in `test.step`, so the
|
|
86
|
+
report and the trace read like the case.
|
|
87
|
+
|
|
88
|
+
- **`- Ready when:`** under a step says what positive signal shows that what the step loads is really there (the list
|
|
89
|
+
has its first row, the editor is visible). The generated spec waits for it before it reads anything. It is not an
|
|
90
|
+
assertion; what must be true goes under `## Expected`.
|
|
91
|
+
- **Step tags**: `[writes]` for a step that creates or changes data, `[input]` for a step that needs a person's
|
|
92
|
+
choice (an existing document, items), `[email]` for a step that sends email. They are written at the end of the
|
|
93
|
+
step line.
|
|
94
|
+
- **`- Recipe (phase):`** under a tagged step records what a person did in the browser during an attended
|
|
95
|
+
exploration: positions instead of names of changeable data (`select items #1 to #3`), `[asrun]` names for created
|
|
96
|
+
data, control labels but no selectors. Several lines per step are normal. A tagged step with no recipe is not
|
|
97
|
+
generated: `/gen-spec` stops with `needs-input`.
|
|
98
|
+
- **Open questions** go in a `🚫 TODO [blocking]` or `❓ TODO [non-blocking]` block under the sentence they are about,
|
|
99
|
+
with the agent's suggestion written above and a last line `- ✍️ Answer:`. Accepting a question means deleting the
|
|
100
|
+
whole block; answering means writing after `Answer:`. The marker `🔍 to be confirmed by QA` / `👁️ not observed`
|
|
101
|
+
marks something seen but not settled.
|
|
102
|
+
|
|
103
|
+
## Expected
|
|
104
|
+
|
|
105
|
+
Bullets: behaviour that must hold for every profile. Profile differences belong in `## Values`, not in prose.
|
|
106
|
+
|
|
107
|
+
## Values
|
|
108
|
+
|
|
109
|
+
The exact literals the spec asserts, in one fenced `yaml` block: names, labels, options, test data names.
|
|
110
|
+
|
|
111
|
+
```yaml
|
|
112
|
+
default:
|
|
113
|
+
createButton: Create New
|
|
114
|
+
emptyList: No documents yet
|
|
115
|
+
example-staging:
|
|
116
|
+
createButton: Create a document
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
- `default` applies to every profile; an entry named after a profile id is **deep-merged over it**. An empty cell in
|
|
120
|
+
the UI removes the key from that profile.
|
|
121
|
+
- Only profile ids from `profiles.yaml` (or `default`) may appear at the top level. An unknown entry is an error.
|
|
122
|
+
- The case document is the only place where expected values live. `profiles.yaml` holds connection settings only.
|
|
123
|
+
- A profile with no entry (and no `default`) fails with "Missing values"; an empty value or one that starts with `TODO`
|
|
124
|
+
fails with "Incomplete values". Both messages say what to fill in.
|
|
125
|
+
- Specs read them with `requireValues('<id>', profile)` and never parse the YAML themselves.
|
|
126
|
+
|
|
127
|
+
## Visual checks and targets
|
|
128
|
+
|
|
129
|
+
A visual check names the element that must match a screenshot baseline. A check on a logo, image, icon, file, PDF,
|
|
130
|
+
screenshot or repeated text refers to a card under `## Targets` as `[T1]`, and the spec carries `// target: T1` on the
|
|
131
|
+
line before the check.
|
|
132
|
+
|
|
133
|
+
Each card has these fields:
|
|
134
|
+
|
|
135
|
+
- `What`: the thing itself, and what it is **not** when a look-alike exists.
|
|
136
|
+
- `Where`: the region of the page, dialog, tab or file page.
|
|
137
|
+
- `Identified by`: role and name, test id, alt text or title; never a position.
|
|
138
|
+
- `Not`: the look-alikes that must not be matched.
|
|
139
|
+
- `Ready when`: what shows the content has loaded.
|
|
140
|
+
- `Reference <profile-id>:` a screenshot under `.explore/targets/<case-id>/`, or `none: <reason>`.
|
|
141
|
+
|
|
142
|
+
Do not guess a target: when it is unclear which element is meant, the exploring agent writes a `🚫 TODO [blocking]`
|
|
143
|
+
that names the candidates, and a person settles it. The binding rules are in
|
|
144
|
+
`../.claude/skills/exploring-cases/references/assertion-targets.md` of the framework package.
|
|
145
|
+
|
|
146
|
+
## Data writing
|
|
147
|
+
|
|
148
|
+
By default a case is read-only: look, read, compare. No creating, editing, deleting, submitting or accepting
|
|
149
|
+
anything.
|
|
150
|
+
|
|
151
|
+
A case may write only in the environments listed under `writes`, and only data of the test account:
|
|
152
|
+
|
|
153
|
+
- everything it creates is named `[asrun] ...` (the `records` fixture refuses other names);
|
|
154
|
+
- created data is registered with `records.track` and removed after the test, also after a failure;
|
|
155
|
+
- the case's own last step is normally the deletion; `track` is the safety net;
|
|
156
|
+
- staging needs no confirmation. In any other environment a person confirms every run (typing the environment name,
|
|
157
|
+
or the dialog in the web UI). `--read-only` leaves the write cases out, and `--allow-writes=<env>` skips the prompt
|
|
158
|
+
for unattended runs (AI agents never use it);
|
|
159
|
+
- write cases are never retried and run one at a time per profile;
|
|
160
|
+
- a case that is not allowed to write in the current environment is **skipped with a notice**, not failed;
|
|
161
|
+
- whatever could not be removed is listed in red in the report and by `pnpm cleanup --env <env>`; delete it by hand,
|
|
162
|
+
then `--forget`.
|
|
163
|
+
|
|
164
|
+
Email is never sent in production, and elsewhere only when the profile has `email: true`. A step that sends email is
|
|
165
|
+
wrapped in the spec with `emailStepAllowed(profile, '<step title>')`, which records the skip in the report.
|
|
166
|
+
|
|
167
|
+
Never write a case for the login page or the login flow: login is a person's job (`pnpm signin`), not a test.
|
|
168
|
+
|
|
169
|
+
## What `pnpm case:check` validates
|
|
170
|
+
|
|
171
|
+
It reads every case (or the ids you pass) and prints one line per case. It **fails** on:
|
|
172
|
+
|
|
173
|
+
- invalid front matter (missing `suites`, a bad `writes`/`envs` combination, `writes` without `writes-note`);
|
|
174
|
+
- a target card without its required fields (`What`, `Where`, `Identified by`), or a reference image outside
|
|
175
|
+
`.explore/targets/<case-id>/`;
|
|
176
|
+
- a `[T<n>]` reference with no card;
|
|
177
|
+
- an invalid `## Values` block (bad YAML, an unknown profile, a non-map entry).
|
|
178
|
+
|
|
179
|
+
It **warns** (usable, but something to settle) on: a reference image missing on this machine (`.explore/` is
|
|
180
|
+
git-ignored), a card without a reference line, a check on an image or file with no target card, no `## Values`, a
|
|
181
|
+
`[input]`/`[writes]` step with no `- Recipe:` line, the old `## Inputs` / `## Data` sections, and question blocks
|
|
182
|
+
that cannot be answered in place.
|
|
183
|
+
|
|
184
|
+
## Rename or move a case
|
|
185
|
+
|
|
186
|
+
Use the Cases page (click the file name or the pencil icon). The spec and the baseline images move with it and the
|
|
187
|
+
spec is rewritten to match. By hand, move the three things together (`cases/...md`, `tests/...spec.ts`, its
|
|
188
|
+
`*-snapshots/`), and keep the spec's `// case:` line and its imports unchanged: specs import the framework by package
|
|
189
|
+
name, so their folder depth does not change the imports.
|