@asrunjs/asrun 0.2.0 → 0.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/explore-case.md +18 -15
- package/.claude/commands/gen-spec.md +16 -13
- package/.claude/commands/heal-spec.md +26 -23
- package/.claude/commands/make-case.md +11 -8
- package/.claude/skills/exploring-cases/SKILL.md +146 -132
- package/.claude/skills/exploring-cases/references/open-questions.md +116 -112
- package/.claude/skills/exploring-cases/references/readiness.md +146 -137
- package/.claude/skills/exploring-cases/references/user-handoff.md +157 -147
- package/.claude/skills/generating-specs/SKILL.md +219 -207
- package/.claude/skills/generating-specs/references/conventions.md +148 -148
- package/.claude/skills/making-cases/SKILL.md +72 -64
- package/.claude/skills/making-cases/references/normalizing.md +39 -38
- package/.claude/skills/making-cases/references/writes-and-email.md +37 -36
- package/AGENTS.md +169 -154
- package/CLAUDE.md +2 -1
- package/README.md +221 -410
- package/dist/lib/build-artifacts.d.ts +8 -0
- package/dist/lib/build-artifacts.js +10 -0
- package/dist/lib/case-doc.d.ts +311 -0
- package/dist/lib/case-doc.js +1117 -0
- package/dist/lib/cases.d.ts +2 -15
- package/dist/lib/cases.js +5 -78
- package/dist/lib/command.d.ts +24 -0
- package/dist/lib/command.js +90 -0
- package/dist/lib/fixtures.js +2 -2
- package/dist/lib/login-card.d.ts +22 -0
- package/dist/lib/login-card.js +183 -0
- package/dist/lib/node-cli.d.ts +6 -2
- package/dist/lib/node-cli.js +16 -6
- package/dist/lib/profiles.d.ts +25 -1
- package/dist/lib/profiles.js +54 -1
- package/dist/lib/run-filters.js +10 -2
- package/dist/lib/session.d.ts +13 -1
- package/dist/lib/session.js +73 -7
- package/dist/lib/targets.d.ts +5 -13
- package/dist/lib/targets.js +4 -29
- package/dist/lib/values.js +2 -2
- package/dist/scripts/build.js +2 -1
- package/dist/scripts/check-cases.js +74 -6
- package/dist/scripts/doctor.js +115 -4
- package/dist/scripts/explore.js +22 -1
- package/dist/scripts/go.mjs +188 -188
- package/dist/scripts/init.d.ts +7 -1
- package/dist/scripts/init.js +62 -14
- package/dist/scripts/lint-specs.js +4 -1
- package/dist/scripts/login.js +30 -2
- package/dist/ui/server/actions.d.ts +4 -0
- package/dist/ui/server/actions.js +25 -12
- package/dist/ui/server/ai-models.d.ts +3 -0
- package/dist/ui/server/ai-models.js +347 -0
- package/dist/ui/server/ai-stream.d.ts +34 -9
- package/dist/ui/server/ai-stream.js +283 -77
- package/dist/ui/server/app.js +74 -12
- package/dist/ui/server/jobs.d.ts +7 -1
- package/dist/ui/server/jobs.js +69 -15
- package/dist/ui/server/profiles.d.ts +20 -1
- package/dist/ui/server/profiles.js +214 -38
- package/dist/ui/shared/ai-commands.d.ts +16 -6
- package/dist/ui/shared/ai-commands.js +35 -12
- package/dist/ui/shared/types.d.ts +114 -0
- package/docs/agents/case-format.md +193 -0
- package/docs/agents/framework-dev.md +167 -0
- package/docs/ai-workflow.md +235 -0
- package/docs/cases.md +189 -0
- package/docs/commands.md +175 -0
- package/docs/install.md +174 -0
- package/docs/profiles.md +139 -0
- package/docs/theme-guide.html +922 -692
- package/docs/troubleshooting.md +101 -0
- package/docs/web-ui.md +139 -0
- package/package.json +20 -1
- package/templates/.claude/commands/explore-case.md +22 -21
- package/templates/.claude/commands/gen-spec.md +21 -20
- package/templates/.claude/commands/heal-spec.md +31 -30
- package/templates/.claude/commands/make-case.md +13 -12
- package/templates/.mcp.json +8 -8
- package/templates/AGENTS.md +66 -45
- package/templates/cases/_template.md +57 -63
- package/templates/opencode.jsonc +16 -11
- package/templates/profiles.yaml +42 -29
- package/ui/dist/assets/{dist-D_I45xgJ.js → dist-BudNAjOv.js} +1 -1
- package/ui/dist/assets/{dist-CBV7_vDN.js → dist-C-eCSFja.js} +1 -1
- package/ui/dist/assets/{dist-gF2bFn-5.js → dist-CO4heSzh.js} +1 -1
- package/ui/dist/assets/{dist-DX8knLVr.js → dist-CPLhRzom.js} +1 -1
- package/ui/dist/assets/{dist-BqAROxzI.js → dist-DEPJZ3Me.js} +1 -1
- package/ui/dist/assets/{dist-BwYRbe5s.js → dist-DZ7RiqmA.js} +1 -1
- package/ui/dist/assets/{dist-BHaIOZdW.js → dist-LixzGniF.js} +1 -1
- package/ui/dist/assets/{dist-DHCY9KpV.js → dist-O_RVi4tP.js} +1 -1
- package/ui/dist/assets/{dist-BiE4suo9.js → dist-jPS6HCJ5.js} +1 -1
- package/ui/dist/assets/{dist-BSJMM_jp.js → dist-vmf3fnkM.js} +1 -1
- package/ui/dist/assets/index-BIGt8Hfz.js +338 -0
- package/ui/dist/assets/index-CTmnF0Wp.css +2 -0
- package/ui/dist/help/ai-tools.png +0 -0
- package/ui/dist/help/case-explore.png +0 -0
- package/ui/dist/help/case-generate.png +0 -0
- package/ui/dist/help/cases-memo.png +0 -0
- package/ui/dist/help/profile-form.png +0 -0
- package/ui/dist/help/profiles-login.png +0 -0
- package/ui/dist/help/reports.png +0 -0
- package/ui/dist/help/run-tests.png +0 -0
- package/ui/dist/index.html +22 -16
- package/ui/dist/assets/index-DmasQ6Rq.css +0 -2
- package/ui/dist/assets/index-I7uU93kk.js +0 -294
|
@@ -1,207 +1,219 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: generating-specs
|
|
3
|
-
description: Turns a human-written test case (cases/<id>.md, the id may include a group folder such as documents/landing-page) into a Playwright spec (tests/<id>.spec.ts) for the multi-profile test suite, explores the real pages to learn the locators and values, and completes the case's own "## Values" block. It only generates: it never runs the tests. Use when asked to generate, update, fix or heal a spec, or for /gen-spec.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Generating specs from case documents
|
|
7
|
-
|
|
8
|
-
Read `references/conventions.md` first. It defines the spec shape, fixtures, assertion layers and
|
|
9
|
-
the read-only rules. Read `AGENTS.md` for hard rules. Read `../exploring-cases/references/open-questions.md`: it
|
|
10
|
-
defines how open questions are written into the case file, resolved and reported. Also read
|
|
11
|
-
`../exploring-cases/references/assertion-targets.md` (the exact thing each check looks at) and
|
|
12
|
-
`../exploring-cases/references/readiness.md` (wait for the content, verify downloads in five parts). All three are binding here.
|
|
13
|
-
|
|
14
|
-
This skill never stops to ask the user in the chat, because it may run without anybody present (a one-shot
|
|
15
|
-
`claude -p` / `opencode run`). Questions go into the case file; the final line is `RESULT: ready` or
|
|
16
|
-
`RESULT: needs-input`.
|
|
17
|
-
|
|
18
|
-
This skill trusts the case. What the pages show, what a person has to enter, and what the open questions are is
|
|
19
|
-
settled by `/explore-case` and recorded in the case (`(observed, ...)` marks, the
|
|
20
|
-
only finds locators and writes code: it replays the case and does not judge the case again.
|
|
21
|
-
|
|
22
|
-
Generating a spec and running it are separate processes. This skill only writes files: it does not run
|
|
23
|
-
`pnpm asrun`, does not create screenshot baselines and does not judge whether the spec passes. The user runs the
|
|
24
|
-
spec afterwards (see the report). Healing a failing spec (below) is the exception: it reads the last report first and runs only the profiles that are logged in, and only when the report is not enough.
|
|
25
|
-
|
|
26
|
-
##
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
`
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
-
|
|
70
|
-
|
|
71
|
-
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
that
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
`
|
|
123
|
-
|
|
124
|
-
(
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
(
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
`
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
`--
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
1
|
+
---
|
|
2
|
+
name: generating-specs
|
|
3
|
+
description: Turns a human-written test case (cases/<id>.md, the id may include a group folder such as documents/landing-page) into a Playwright spec (tests/<id>.spec.ts) for the multi-profile test suite, explores the real pages to learn the locators and values, and completes the case's own "## Values" block. It only generates: it never runs the tests. Use when asked to generate, update, fix or heal a spec, or for /gen-spec.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Generating specs from case documents
|
|
7
|
+
|
|
8
|
+
Read `references/conventions.md` first. It defines the spec shape, fixtures, assertion layers and
|
|
9
|
+
the read-only rules. Read the workspace `AGENTS.md` for the hard rules and the project rules. Read `../exploring-cases/references/open-questions.md`: it
|
|
10
|
+
defines how open questions are written into the case file, resolved and reported. Also read
|
|
11
|
+
`../exploring-cases/references/assertion-targets.md` (the exact thing each check looks at) and
|
|
12
|
+
`../exploring-cases/references/readiness.md` (wait for the content, verify downloads in five parts). All three are binding here.
|
|
13
|
+
|
|
14
|
+
This skill never stops to ask the user in the chat, because it may run without anybody present (a one-shot
|
|
15
|
+
`claude -p` / `opencode run`). Questions go into the case file; the final line is `RESULT: ready` or
|
|
16
|
+
`RESULT: needs-input`.
|
|
17
|
+
|
|
18
|
+
This skill trusts the case. What the pages show, what a person has to enter, and what the open questions are is
|
|
19
|
+
settled by `/explore-case` and recorded in the case (`(observed, ...)` marks, the step recipes). Generating
|
|
20
|
+
only finds locators and writes code: it replays the case and does not judge the case again.
|
|
21
|
+
|
|
22
|
+
Generating a spec and running it are separate processes. This skill only writes files: it does not run
|
|
23
|
+
`pnpm asrun`, does not create screenshot baselines and does not judge whether the spec passes. The user runs the
|
|
24
|
+
spec afterwards (see the report). Healing a failing spec (below) is the exception: it reads the last report first and runs only the profiles that are logged in, and only when the report is not enough.
|
|
25
|
+
|
|
26
|
+
## Before you start
|
|
27
|
+
Read the workspace `AGENTS.md`: its §2 holds the project rules, and it points at the framework baseline. If
|
|
28
|
+
`.claude/skills/generating-specs.md` exists in the workspace, read it and apply it after this skill. Then apply
|
|
29
|
+
`AGENTS.md` §2, which wins over both (the baseline, "Precedence"). F1–F6 always win. This is not a stop condition:
|
|
30
|
+
continue with the procedure below.
|
|
31
|
+
|
|
32
|
+
## Inputs
|
|
33
|
+
- The case id and the environment to explore in (`staging` or `production`). There is no default. If the
|
|
34
|
+
environment is missing, stop before exploring and report `RESULT: needs-input` ("pass the environment").
|
|
35
|
+
- Optionally `--force`: regenerate the spec from scratch and overwrite the existing one. See "An existing spec".
|
|
36
|
+
|
|
37
|
+
A case id is its path below `cases/` without `.md`: `hub-landing`, or `documents/landing-page` when the case is in a
|
|
38
|
+
folder (one level only). The spec is at the same path below `tests/` (`tests/documents/landing-page.spec.ts`); both import
|
|
39
|
+
the framework through the `@asrunjs/asrun/*` alias, so the folder depth does not change the imports.
|
|
40
|
+
|
|
41
|
+
- `cases/<id>.md` (front matter: `id`, `login: required|none` (`none` = public page; the login page itself is not a valid case), `suites: [<name>, ...]` (the old `priority: <level>` means one suite), `writes: [<env>, ...]` (omitted = read-only; the environments the spec may write in, e.g. `[staging]`) + `writes-note`, optional `envs: [...]`; sections Steps (with a step's `- Recipe:` lines) / Expected / Values / Visual checks / Targets; an old `## Inputs` section may still hold recipes). `## Values` holds one fenced yaml block keyed by `default` and by profile id, with the exact literals the spec asserts. The case document is the only place where expected values live.
|
|
42
|
+
- `profiles.yaml` (profiles: id, name, env, baseURL, account, tags, email). It holds no expected values.
|
|
43
|
+
|
|
44
|
+
## An existing spec
|
|
45
|
+
Decide this first, before exploring anything. The spec path is `tests/<id>.spec.ts`.
|
|
46
|
+
- **No spec yet:** generate it (the procedure below).
|
|
47
|
+
- **A spec exists and `--force` was not given:** do not touch it. Do not edit the spec, the case values or any baseline.
|
|
48
|
+
Report that the spec already exists, say that `--force` regenerates and overwrites it, and give the id and
|
|
49
|
+
environment to run it again with. End with `RESULT: ready` (the spec that exists is not in question). Nothing else
|
|
50
|
+
is written, not even the case file.
|
|
51
|
+
- **A spec exists and `--force` was given:** the user chose to overwrite it. Regenerate it from the case with the
|
|
52
|
+
procedure below, and write the new file over the old one (do not merge with it: the case file is the source of
|
|
53
|
+
truth). Do not read the old spec as a starting point and do not keep it just because the result looks the same:
|
|
54
|
+
write the file from the case and the exploration. If it ends up identical, say so in the report. Keep the `## Values`
|
|
55
|
+
of other cases untouched; only this case's `## Values` is yours. Leave the
|
|
56
|
+
baseline images in `tests/<id>.spec.ts-snapshots/` alone: a changed visual check needs a person to update them,
|
|
57
|
+
so list in the report which baselines may now be out of date. Say in the report that an existing spec was
|
|
58
|
+
overwritten.
|
|
59
|
+
|
|
60
|
+
## Procedure
|
|
61
|
+
1. **Make sure the case uses the artifact language.** `AGENTS.md` §2 sets it (English by default), and spec step
|
|
62
|
+
titles copy the case's step text. If `cases/<id>.md` is written in another language, translate it into that
|
|
63
|
+
language and rewrite the file BEFORE anything else: keep the front matter keys and values, the section headings,
|
|
64
|
+
the `## Values` entries and the step numbering as they are, and keep quoted UI text exactly as it appears
|
|
65
|
+
on the page (do not translate strings the page shows). Do not add, drop or reinterpret any step or
|
|
66
|
+
expectation; if a sentence is unclear, translate it literally and raise it as a question in the next step.
|
|
67
|
+
Tell the user in the report that the case file was translated. Skip this when the case already uses that language.
|
|
68
|
+
2. **Validate the case.** Collect everything that is ambiguous, every check without an expected value and every
|
|
69
|
+
missing per-profile value, including the `TODO` items already in the case. Resolve them in the order of
|
|
70
|
+
`open-questions.md`; do not stop here:
|
|
71
|
+
- **Steps that write or need a choice (`[input]`, `[writes]`) need a recipe.** Each must have at least one
|
|
72
|
+
`- Recipe (phase): ...` line under it in the case (written by `/explore-case`). An old case may still hold the
|
|
73
|
+
same recipe in `## Inputs`: use that line until it is moved under the step. If one has none, the case was
|
|
74
|
+
never explored with a person: stop before opening a browser, do not write the spec, and report `RESULT:
|
|
75
|
+
needs-input` with exactly this: "step <n> has no recorded input; run `/explore-case <id> <env> --attended`
|
|
76
|
+
first". Nothing else is written.
|
|
77
|
+
- Facts the page can answer and that the case does not hold yet are read in step 4 and written into the case as
|
|
78
|
+
`(observed, 🔍 to be confirmed by QA)`. A spec is generated from observed values.
|
|
79
|
+
- Safe defaults are applied: the suggestion is the sentence above the TODO block, and the block stays in the case
|
|
80
|
+
as `❓ TODO [non-blocking]` with an empty `✍️ Answer:`. A sentence with no TODO block under it is confirmed text.
|
|
81
|
+
- A non-empty `Answer:` is the user's answer: it wins over the sentence above it. Fold it into the case.
|
|
82
|
+
- Only what is always-blocking in `open-questions.md` blocks. Generate everything else; for a blocked check
|
|
83
|
+
(a `[blocking]` block), write `test.fixme(true, '<the question>')` inside its step (or skip that profile), never
|
|
84
|
+
an invented value and never a check built from the sentence above a `[blocking]` block.
|
|
85
|
+
- **Every check on a logo, image, icon, file, PDF, screenshot or repeated text needs a target card** (`## Targets`,
|
|
86
|
+
referenced as `[T<n>]`) that was observed (`Reference <profile-id>:` is a screenshot, not `none`) and has no open
|
|
87
|
+
`[blocking]` block. A check that has no card, or whose card is not observed, is not guessed here. Do not pick
|
|
88
|
+
an element yourself: report `RESULT: needs-input` with "check <text> has no observed target; run `/explore-case
|
|
89
|
+
<id> <env>` first" (or, when the card has a `[blocking]` TODO, write `test.fixme(true, '<the question>')` for that check as
|
|
90
|
+
the rule above says and go on with the rest).
|
|
91
|
+
- **Evidence gate:** a hard assertion (one that decides pass or fail) on something nobody has seen on a page is not
|
|
92
|
+
written. "Not verified on the page" is allowed for a locator of a page that only exists after a write step, but
|
|
93
|
+
never for what the check asserts about it: that goes to `test.fixme` with the reason, or `needs-input`.
|
|
94
|
+
- Do not invent expected values. If the case's `## Values` and the spec disagree, the case wins: fix the spec and
|
|
95
|
+
mention the change in the report.
|
|
96
|
+
3. **Check sessions** if `login: required`: run `pnpm status --env <env>`. If a profile needs login, tell the
|
|
97
|
+
user to run `pnpm signin <profile-id>` and continue with the profiles that are ready. If `pnpm status`
|
|
98
|
+
cannot run (a sandbox error such as `listen EPERM` or a Chromium start failure), judge by the page instead: a page
|
|
99
|
+
that lands on the login page means the profile is not logged in.
|
|
100
|
+
4. **Replay the case on the real page** with the `playwright-test` MCP. If its tools are not registered in this
|
|
101
|
+
session (no `browser_*` / `generator_*` tool) or it reports that it failed to start, do not fall back to ad-hoc
|
|
102
|
+
Playwright scripts: stop, tell the user which MCP server is missing and to run `pnpm asrun doctor`, write no spec,
|
|
103
|
+
and end with `RESULT: needs-input`.
|
|
104
|
+
Call `generator_setup_page` with
|
|
105
|
+
`project=<profile-id>` (for example `example-staging`) and `seedFile=tests/seed.spec.ts`, then perform the
|
|
106
|
+
steps in the environment you were given (never ask for it; there is no default) to learn the locators.
|
|
107
|
+
Do the read-only steps as written. **Never perform a step that writes or needs a choice**: the agent does not write
|
|
108
|
+
data, in staging or in production, and it does not hand off to a person here or wait for one. Write those steps
|
|
109
|
+
of the spec from the `- Recipe:` lines under the step (positions, `[asrun]` names, control labels and roles that
|
|
110
|
+
`/explore-case` recorded). Pages that only exist after such a step cannot be seen from here: write their locators
|
|
111
|
+
from what the recipe and the exploration notes say, mark them in the report as "not verified on the page", and
|
|
112
|
+
leave the first run of the spec (or `/heal-spec`) to confirm them. Do this for at least one profile
|
|
113
|
+
per domain (example.com and exampleshop.com differ in markup). Profiles that are not logged in are skipped and
|
|
114
|
+
named in the report. Use this exploration to answer the open points of step 2, and look at every profile that has
|
|
115
|
+
an entry in `## Values` to read its value.
|
|
116
|
+
Keep every browser call short (one action, about 60 s at most, a snapshot between steps, `{timeout: 5000}` when
|
|
117
|
+
something may not appear, no third retry): see "Keep every browser call short" in `../exploring-cases/SKILL.md`.
|
|
118
|
+
**Ready before you read, here too** (`readiness.md`): `browser_network_clear`, act, wait for the key requests
|
|
119
|
+
(`browser_network_requests`) and the content signal, then snapshot. The request URL patterns for `actAndWait` and the
|
|
120
|
+
`Ready when` signals of the case are taken from what you see here.
|
|
121
|
+
**Check every target on the page:** for each target card, find its `Identified by` locator, confirm that it matches
|
|
122
|
+
exactly one element and that this element is what `What` says (highlight it and read the screenshot). A locator that
|
|
123
|
+
matches several elements or the wrong one is fixed or becomes a question, never `.first()`.
|
|
124
|
+
Prefer `getByTestId` (the project uses `data-tn`), then role/label locators. No CSS chains or XPath.
|
|
125
|
+
Update the case file with what you learned: fold observations into the right section marked
|
|
126
|
+
`(observed, 🔍 to be confirmed by QA)`, remove the TODO blocks they answer (rewrite the sentence above the block),
|
|
127
|
+
and narrow the ones that remain. Keep the shape of `open-questions.md` for every block. Edit the case
|
|
128
|
+
with targeted edits; never touch its `## Source` table.
|
|
129
|
+
5. **Complete the case's `## Values`**: for every profile the spec runs for, add the entry with the literals the spec
|
|
130
|
+
asserts (the same keys as `default`, only what differs). Do not move values into `profiles.yaml`; it holds connection
|
|
131
|
+
settings only. Mention every value you added or corrected in the report.
|
|
132
|
+
6. **Write `tests/<id>.spec.ts`** (inside the group folder if the id has one, creating it if needed) following the conventions. One `test()` per outcome, and every
|
|
133
|
+
case step wrapped in `test.step()` with the case's step text as the title, and the top-level
|
|
134
|
+
`test.describe` tagged with `caseTags('<id>')`. The spec must
|
|
135
|
+
contain no profile-specific literals: everything profile-specific comes from `requireValues('<id>', profile)`
|
|
136
|
+
(`@asrunjs/asrun/values`), which merges `default` with the running profile's entry.
|
|
137
|
+
Every action that loads something goes through `actAndWait` (`@asrunjs/asrun/ready`) with the key requests and the ready
|
|
138
|
+
signal; every download goes through `verifyDownload` (`@asrunjs/asrun/downloads`); every check on a target has
|
|
139
|
+
`// target: T<n>` on the line before it; every absence check has a `// ready: <signal>` before it. Never
|
|
140
|
+
`waitForTimeout`, `networkidle` or `.first()` to pick an image. A confirmation or modal that may or may not appear is
|
|
141
|
+
never decided by one `isVisible()` look (its `timeout` option is ignored): use `clickOptionalUntilHidden`
|
|
142
|
+
(`@asrunjs/asrun/ready`) or a waiting `expect(...).toBeVisible({timeout})`; see `readiness.md`, "Optional confirmations".
|
|
143
|
+
7. **Check the files**: run `pnpm typecheck`, `pnpm spec:lint <id>` and `pnpm case:check` (or `node --import tsx <framework>/scripts/check-cases.ts` if
|
|
144
|
+
it fails with a sandbox error such as `listen EPERM`). Fix every error, including every `FAIL` of the spec lint (a lint failure is never silenced: fix the spec or turn the check
|
|
145
|
+
into a question). Every lint `WARN` is handled too: fix it, or name it in the report with the reason it is acceptable.
|
|
146
|
+
Then do the **self-review**: for each assertion answer in writing "if the product were broken, would this fail?"
|
|
147
|
+
and "if the product is right, could this fail by timing?" (a check that cannot fail, or that races the page, is rewritten).
|
|
148
|
+
For every branch that depends on something possibly not there yet (`if`, `.count()`, `.catch(() => false)`), ask "what
|
|
149
|
+
if it shows up a second later?"; the answer must be a retry loop or a waiting assertion, never a single look.
|
|
150
|
+
Do NOT run `pnpm asrun`, do not use
|
|
151
|
+
`--update-snapshots` and do not create baselines: the first run of the spec does that, and a person reviews them.
|
|
152
|
+
8. **Report** in the user's language: spec path, **an assertion table** (step, what is asserted, target card, ready signal, and
|
|
153
|
+
`observed on the page` or `not verified`; no row may be missing), profiles explored, profiles skipped (and why), whether the case file was
|
|
154
|
+
translated, what was added to the case file as observed, checks generated from observations (QA must confirm
|
|
155
|
+
them), visual checks that need a baseline on the first run, and the remaining TODO blocks (`[blocking]` ones with
|
|
156
|
+
the sentence that deleting the block would confirm). State that the spec was not run and give the command to run it:
|
|
157
|
+
`pnpm asrun --env <env> --profile <ids> --grep <id>` (with `--update-snapshots` once if the spec has visual
|
|
158
|
+
checks, then run `pnpm baselines:review <id> --new` and open every new image: it must show what its target card says). The last line is `RESULT: ready` (no `[blocking]` question left, spec
|
|
159
|
+
written and the checks above pass) or `RESULT: needs-input`, as defined in `open-questions.md`.
|
|
160
|
+
|
|
161
|
+
## Cases that write data
|
|
162
|
+
Only when the case front matter lists environments under `writes` (for example `writes: [staging]`). Never add
|
|
163
|
+
`--allow-writes` or set `ASRUN_ALLOW_WRITES`. Use the `records` fixture, `configureCase` and `emailStepAllowed` as
|
|
164
|
+
described in `references/conventions.md`. The agent does not perform write steps while generating (see step 4); the
|
|
165
|
+
spec does, when a person runs it in an environment that `writes` lists. Elsewhere the runner skips the case with a
|
|
166
|
+
notice. Never send email in production; in other environments only if `profile.email` is true, otherwise skip that step and say so
|
|
167
|
+
in the report.
|
|
168
|
+
|
|
169
|
+
## Healing a failing spec
|
|
170
|
+
Inputs: the case id and the environment (both required, there is no default: without an environment stop and say so),
|
|
171
|
+
optionally `--profiles a,b`, `--report <run-id>` and `--hints <file>` (or free text after the arguments). Healing is
|
|
172
|
+
the one case where this skill runs specs, and only in the environment given.
|
|
173
|
+
|
|
174
|
+
Heal from evidence, not from a habit of rerunning. A report that already exists says what failed.
|
|
175
|
+
|
|
176
|
+
1. **Read the user's clues first** (the `--hints` file, or the free text). They say what to look at and may hold
|
|
177
|
+
evidence the report lacks. They guide the diagnosis and nothing else: they never lower the bar for a product
|
|
178
|
+
finding and never allow an expectation to be changed just to make a test pass. Quote the clue you used, or say that
|
|
179
|
+
it did not hold up, in the report.
|
|
180
|
+
2. **Read the existing report**: run `pnpm heal:evidence --case <case-id> --env <env>` (add `--profile a,b` when `--profiles`
|
|
181
|
+
was given). It reads `reports/*/summary.json` and prints, per profile, the status, the failed step, the error, the call
|
|
182
|
+
log, the screenshots and videos, and ends with `EVIDENCE: fresh | stale | no-failure | none`. With `--report <run-id>`,
|
|
183
|
+
read `reports/<run-id>/summary.json` instead. Look at the screenshot when the error alone does not explain it. A profile
|
|
184
|
+
that failed only because it was not logged in is no evidence about the spec.
|
|
185
|
+
3. **Decide whether to rerun the case**. Do not rerun when the report is `fresh`, names the failing step and the error,
|
|
186
|
+
and that is enough to tell whether the product changed or the test is wrong. Rerun when:
|
|
187
|
+
- there is no report (`none`), or it is `stale` (the spec, the case or `profiles.yaml` changed after it);
|
|
188
|
+
- the report does not explain the failure (a bare timeout with no locator, a failure before any step, a truncated log);
|
|
189
|
+
- a clue from the user has to be checked against the real page;
|
|
190
|
+
- a profile that must be healed has no result in the report (for example it was not logged in then).
|
|
191
|
+
Rerun with one plain command, as few profiles as possible, and say why in the report:
|
|
192
|
+
`pnpm asrun --env <env> --logged-in --profile <ids> --grep <case-id>` (optionally piped to `tail`; do not chain it
|
|
193
|
+
with other commands, or it runs inside Claude's sandbox and fails with `listen EPERM`). Always add `--logged-in`: it
|
|
194
|
+
leaves out the profiles that are not logged in instead of failing them. A profile that is not logged in is never healed:
|
|
195
|
+
name it in the report with `pnpm signin <profile-id>`.
|
|
196
|
+
4. **Audit the assertion before you blame anyone.** Read the `assertion-targets.md` and `readiness.md` sections on healing.
|
|
197
|
+
- Is the locator still what the target card says (`What`, `Identified by`)? Open the failure screenshot and the
|
|
198
|
+
baseline (`pnpm baselines:review <case-id>`) and look at each: a baseline that shows something other than the card
|
|
199
|
+
(a content image instead of a logo) is a wrong test, and passing runs on it prove nothing. Delete it and report it; a
|
|
200
|
+
person creates the new baseline after the spec is fixed. Note: other profiles that pass on the same wrong target are also suspect.
|
|
201
|
+
- Is the failure a "missing / empty / not visible / timeout" kind? Then it is first a readiness suspect: was the page
|
|
202
|
+
still loading in the screenshot or video? Fix with `actAndWait` and a ready signal, never with a longer timeout alone or a sleep.
|
|
203
|
+
- Is the spec weak in the ways the lint knows? Run `pnpm spec:lint <case-id>` and fix every `FAIL` while you are there.
|
|
204
|
+
- A download that fails says which stage failed (start, finish, exists, valid): fix the stage, do not loosen the check.
|
|
205
|
+
- Doubt about which element is meant: write the question into the case (a `[blocking]` TODO naming the exact image,
|
|
206
|
+
text, place or UI) instead of choosing; do not rewrite the card quietly.
|
|
207
|
+
5. **Decide whether the PRODUCT changed or the TEST is wrong.**
|
|
208
|
+
- Locator/timing problem: fix the spec.
|
|
209
|
+
- A real product difference (wrong logo, wrong name): do NOT change the expectation to make it
|
|
210
|
+
pass. Report it as a finding with the trace/screenshot path.
|
|
211
|
+
6. **After editing the spec, run it once to confirm** (same command as in step 3, for the profiles you healed). That rerun
|
|
212
|
+
is the only one that is always allowed. Run `pnpm typecheck` too.
|
|
213
|
+
7. **Report** in the user's language: which report was read (or that there was none) and whether it was fresh or stale,
|
|
214
|
+
whether the case was run again and why (or why not), what was changed, what is a product finding (with paths),
|
|
215
|
+
which assertions were wrong or raced the page, which baselines are wrong and were deleted, what the user's clues contributed, which profiles were left out because they are not logged in, and the result of the
|
|
216
|
+
confirming run.
|
|
217
|
+
|
|
218
|
+
A case that writes data in `production` needs a person's confirmation at run time, so the rerun is refused there:
|
|
219
|
+
use the report, or heal in staging, and never add `--allow-writes`.
|