@asrunjs/asrun 0.2.0 → 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. package/.claude/commands/explore-case.md +18 -15
  2. package/.claude/commands/gen-spec.md +16 -13
  3. package/.claude/commands/heal-spec.md +26 -23
  4. package/.claude/commands/make-case.md +11 -8
  5. package/.claude/skills/exploring-cases/SKILL.md +146 -132
  6. package/.claude/skills/exploring-cases/references/open-questions.md +116 -112
  7. package/.claude/skills/exploring-cases/references/readiness.md +146 -137
  8. package/.claude/skills/exploring-cases/references/user-handoff.md +157 -147
  9. package/.claude/skills/generating-specs/SKILL.md +219 -207
  10. package/.claude/skills/generating-specs/references/conventions.md +148 -148
  11. package/.claude/skills/making-cases/SKILL.md +72 -64
  12. package/.claude/skills/making-cases/references/normalizing.md +39 -38
  13. package/.claude/skills/making-cases/references/writes-and-email.md +37 -36
  14. package/AGENTS.md +169 -154
  15. package/CLAUDE.md +2 -1
  16. package/README.md +221 -410
  17. package/dist/lib/build-artifacts.d.ts +8 -0
  18. package/dist/lib/build-artifacts.js +10 -0
  19. package/dist/lib/case-doc.d.ts +311 -0
  20. package/dist/lib/case-doc.js +1117 -0
  21. package/dist/lib/cases.d.ts +2 -15
  22. package/dist/lib/cases.js +5 -78
  23. package/dist/lib/command.d.ts +24 -0
  24. package/dist/lib/command.js +90 -0
  25. package/dist/lib/fixtures.js +2 -2
  26. package/dist/lib/login-card.d.ts +22 -0
  27. package/dist/lib/login-card.js +183 -0
  28. package/dist/lib/node-cli.d.ts +6 -2
  29. package/dist/lib/node-cli.js +16 -6
  30. package/dist/lib/profiles.d.ts +25 -1
  31. package/dist/lib/profiles.js +54 -1
  32. package/dist/lib/run-filters.js +10 -2
  33. package/dist/lib/session.d.ts +13 -1
  34. package/dist/lib/session.js +73 -7
  35. package/dist/lib/targets.d.ts +5 -13
  36. package/dist/lib/targets.js +4 -29
  37. package/dist/lib/values.js +2 -2
  38. package/dist/scripts/build.js +2 -1
  39. package/dist/scripts/check-cases.js +74 -6
  40. package/dist/scripts/doctor.js +115 -4
  41. package/dist/scripts/explore.js +22 -1
  42. package/dist/scripts/go.mjs +188 -188
  43. package/dist/scripts/init.d.ts +7 -1
  44. package/dist/scripts/init.js +62 -14
  45. package/dist/scripts/lint-specs.js +4 -1
  46. package/dist/scripts/login.js +30 -2
  47. package/dist/ui/server/actions.d.ts +4 -0
  48. package/dist/ui/server/actions.js +25 -12
  49. package/dist/ui/server/ai-models.d.ts +3 -0
  50. package/dist/ui/server/ai-models.js +347 -0
  51. package/dist/ui/server/ai-stream.d.ts +34 -9
  52. package/dist/ui/server/ai-stream.js +283 -77
  53. package/dist/ui/server/app.js +74 -12
  54. package/dist/ui/server/jobs.d.ts +7 -1
  55. package/dist/ui/server/jobs.js +69 -15
  56. package/dist/ui/server/profiles.d.ts +20 -1
  57. package/dist/ui/server/profiles.js +214 -38
  58. package/dist/ui/shared/ai-commands.d.ts +16 -6
  59. package/dist/ui/shared/ai-commands.js +35 -12
  60. package/dist/ui/shared/types.d.ts +114 -0
  61. package/docs/agents/case-format.md +193 -0
  62. package/docs/agents/framework-dev.md +167 -0
  63. package/docs/ai-workflow.md +235 -0
  64. package/docs/cases.md +189 -0
  65. package/docs/commands.md +175 -0
  66. package/docs/install.md +174 -0
  67. package/docs/profiles.md +139 -0
  68. package/docs/theme-guide.html +922 -692
  69. package/docs/troubleshooting.md +101 -0
  70. package/docs/web-ui.md +139 -0
  71. package/package.json +20 -1
  72. package/templates/.claude/commands/explore-case.md +22 -21
  73. package/templates/.claude/commands/gen-spec.md +21 -20
  74. package/templates/.claude/commands/heal-spec.md +31 -30
  75. package/templates/.claude/commands/make-case.md +13 -12
  76. package/templates/.mcp.json +8 -8
  77. package/templates/AGENTS.md +66 -45
  78. package/templates/cases/_template.md +57 -63
  79. package/templates/opencode.jsonc +16 -11
  80. package/templates/profiles.yaml +42 -29
  81. package/ui/dist/assets/{dist-D_I45xgJ.js → dist-BudNAjOv.js} +1 -1
  82. package/ui/dist/assets/{dist-CBV7_vDN.js → dist-C-eCSFja.js} +1 -1
  83. package/ui/dist/assets/{dist-gF2bFn-5.js → dist-CO4heSzh.js} +1 -1
  84. package/ui/dist/assets/{dist-DX8knLVr.js → dist-CPLhRzom.js} +1 -1
  85. package/ui/dist/assets/{dist-BqAROxzI.js → dist-DEPJZ3Me.js} +1 -1
  86. package/ui/dist/assets/{dist-BwYRbe5s.js → dist-DZ7RiqmA.js} +1 -1
  87. package/ui/dist/assets/{dist-BHaIOZdW.js → dist-LixzGniF.js} +1 -1
  88. package/ui/dist/assets/{dist-DHCY9KpV.js → dist-O_RVi4tP.js} +1 -1
  89. package/ui/dist/assets/{dist-BiE4suo9.js → dist-jPS6HCJ5.js} +1 -1
  90. package/ui/dist/assets/{dist-BSJMM_jp.js → dist-vmf3fnkM.js} +1 -1
  91. package/ui/dist/assets/index-BIGt8Hfz.js +338 -0
  92. package/ui/dist/assets/index-CTmnF0Wp.css +2 -0
  93. package/ui/dist/help/ai-tools.png +0 -0
  94. package/ui/dist/help/case-explore.png +0 -0
  95. package/ui/dist/help/case-generate.png +0 -0
  96. package/ui/dist/help/cases-memo.png +0 -0
  97. package/ui/dist/help/profile-form.png +0 -0
  98. package/ui/dist/help/profiles-login.png +0 -0
  99. package/ui/dist/help/reports.png +0 -0
  100. package/ui/dist/help/run-tests.png +0 -0
  101. package/ui/dist/index.html +22 -16
  102. package/ui/dist/assets/index-DmasQ6Rq.css +0 -2
  103. package/ui/dist/assets/index-I7uU93kk.js +0 -294
@@ -1,207 +1,219 @@
1
- ---
2
- name: generating-specs
3
- description: Turns a human-written test case (cases/<id>.md, the id may include a group folder such as documents/landing-page) into a Playwright spec (tests/<id>.spec.ts) for the multi-profile test suite, explores the real pages to learn the locators and values, and completes the case's own "## Values" block. It only generates: it never runs the tests. Use when asked to generate, update, fix or heal a spec, or for /gen-spec.
4
- ---
5
-
6
- # Generating specs from case documents
7
-
8
- Read `references/conventions.md` first. It defines the spec shape, fixtures, assertion layers and
9
- the read-only rules. Read `AGENTS.md` for hard rules. Read `../exploring-cases/references/open-questions.md`: it
10
- defines how open questions are written into the case file, resolved and reported. Also read
11
- `../exploring-cases/references/assertion-targets.md` (the exact thing each check looks at) and
12
- `../exploring-cases/references/readiness.md` (wait for the content, verify downloads in five parts). All three are binding here.
13
-
14
- This skill never stops to ask the user in the chat, because it may run without anybody present (a one-shot
15
- `claude -p` / `opencode run`). Questions go into the case file; the final line is `RESULT: ready` or
16
- `RESULT: needs-input`.
17
-
18
- This skill trusts the case. What the pages show, what a person has to enter, and what the open questions are is
19
- settled by `/explore-case` and recorded in the case (`(observed, ...)` marks, the `## Inputs` section). Generating
20
- only finds locators and writes code: it replays the case and does not judge the case again.
21
-
22
- Generating a spec and running it are separate processes. This skill only writes files: it does not run
23
- `pnpm asrun`, does not create screenshot baselines and does not judge whether the spec passes. The user runs the
24
- spec afterwards (see the report). Healing a failing spec (below) is the exception: it reads the last report first and runs only the profiles that are logged in, and only when the report is not enough.
25
-
26
- ## Inputs
27
- - The case id and the environment to explore in (`staging` or `production`). There is no default. If the
28
- environment is missing, stop before exploring and report `RESULT: needs-input` ("pass the environment").
29
- - Optionally `--force`: regenerate the spec from scratch and overwrite the existing one. See "An existing spec".
30
-
31
- A case id is its path below `cases/` without `.md`: `hub-landing`, or `documents/landing-page` when the case is in a
32
- folder (one level only). The spec is at the same path below `tests/` (`tests/documents/landing-page.spec.ts`); both import
33
- the framework through the `@asrunjs/asrun/*` alias, so the folder depth does not change the imports.
34
-
35
- - `cases/<id>.md` (front matter: `id`, `login: required|none` (`none` = public page; the login page itself is not a valid case), `suites: [<name>, ...]` (the old `priority: <level>` means one suite), `writes: [<env>, ...]` (omitted = read-only; the environments the spec may write in, e.g. `[staging]`) + `writes-note`, optional `envs: [...]`; sections Steps / Expected / Values / Visual checks / Targets / Inputs). `## Values` holds one fenced yaml block keyed by `default` and by profile id, with the exact literals the spec asserts. The case document is the only place where expected values live.
36
- - `profiles.yaml` (profiles: id, name, env, baseURL, account, tags, email). It holds no expected values.
37
-
38
- ## An existing spec
39
- Decide this first, before exploring anything. The spec path is `tests/<id>.spec.ts`.
40
- - **No spec yet:** generate it (the procedure below).
41
- - **A spec exists and `--force` was not given:** do not touch it. Do not edit the spec, the case values or any baseline.
42
- Report that the spec already exists, say that `--force` regenerates and overwrites it, and give the id and
43
- environment to run it again with. End with `RESULT: ready` (the spec that exists is not in question). Nothing else
44
- is written, not even the case file.
45
- - **A spec exists and `--force` was given:** the user chose to overwrite it. Regenerate it from the case with the
46
- procedure below, and write the new file over the old one (do not merge with it: the case file is the source of
47
- truth). Do not read the old spec as a starting point and do not keep it just because the result looks the same:
48
- write the file from the case and the exploration. If it ends up identical, say so in the report. Keep the `## Values`
49
- of other cases untouched; only this case's `## Values` is yours. Leave the
50
- baseline images in `tests/<id>.spec.ts-snapshots/` alone: a changed visual check needs a person to update them,
51
- so list in the report which baselines may now be out of date. Say in the report that an existing spec was
52
- overwritten.
53
-
54
- ## Procedure
55
- 1. **Make sure the case is in English.** All files in this repo are English (see `AGENTS.md`), and spec step titles
56
- copy the case's step text. If `cases/<id>.md` is written in another language, translate it to English and
57
- rewrite the file BEFORE anything else: keep the front matter keys and values, the section headings, the
58
- `## Values` entries and the step numbering as they are, and keep quoted UI text exactly as it appears
59
- on the page (do not translate strings the page shows). Do not add, drop or reinterpret any step or
60
- expectation; if a sentence is unclear, translate it literally and raise it as a question in the next step.
61
- Tell the user in the report that the case file was translated. Skip this when the case is already English.
62
- 2. **Validate the case.** Collect everything that is ambiguous, every check without an expected value and every
63
- missing per-profile value, including the `TODO` items already in the case. Resolve them in the order of
64
- `open-questions.md`; do not stop here:
65
- - **Steps that write or need a choice (`[input]`, `[writes]`) need a recipe.** Each must have a bullet in the case's
66
- `## Inputs` section (written by `/explore-case`). If one has none, the case was never explored with a person:
67
- stop before opening a browser, do not write the spec, and report `RESULT: needs-input` with exactly this:
68
- "step <n> has no recorded input; run `/explore-case <id> <env> --attended` first". Nothing else is written.
69
- - Facts the page can answer and that the case does not hold yet are read in step 4 and written into the case as
70
- `(observed, 🔍 to be confirmed by QA)`. A spec is generated from observed values.
71
- - Safe defaults are applied: the suggestion is the sentence above the TODO block, and the block stays in the case
72
- as `❓ TODO [non-blocking]` with an empty `✍️ Answer:`. A sentence with no TODO block under it is confirmed text.
73
- - A non-empty `Answer:` is the user's answer: it wins over the sentence above it. Fold it into the case.
74
- - Only what is always-blocking in `open-questions.md` blocks. Generate everything else; for a blocked check
75
- (a `[blocking]` block), write `test.fixme(true, '<the question>')` inside its step (or skip that profile), never
76
- an invented value and never a check built from the sentence above a `[blocking]` block.
77
- - **Every check on a logo, image, icon, file, PDF, screenshot or repeated text needs a target card** (`## Targets`,
78
- referenced as `[T<n>]`) that was observed (`Reference <profile-id>:` is a screenshot, not `none`) and has no open
79
- `[blocking]` block. A check that has no card, or whose card is not observed, is not guessed here. Do not pick
80
- an element yourself: report `RESULT: needs-input` with "check <text> has no observed target; run `/explore-case
81
- <id> <env>` first" (or, when the card has a `[blocking]` TODO, write `test.fixme(true, '<the question>')` for that check as
82
- the rule above says and go on with the rest).
83
- - **Evidence gate:** a hard assertion (one that decides pass or fail) on something nobody has seen on a page is not
84
- written. "Not verified on the page" is allowed for a locator of a page that only exists after a write step, but
85
- never for what the check asserts about it: that goes to `test.fixme` with the reason, or `needs-input`.
86
- - Do not invent expected values. If the case's `## Values` and the spec disagree, the case wins: fix the spec and
87
- mention the change in the report.
88
- 3. **Check sessions** if `login: required`: run `pnpm status --env <env>`. If a profile needs login, tell the
89
- user to run `pnpm signin <profile-id>` and continue with the profiles that are ready. If `pnpm status`
90
- cannot run (a sandbox error such as `listen EPERM` or a Chromium start failure), judge by the page instead: a page
91
- that lands on the login page means the profile is not logged in.
92
- 4. **Replay the case on the real page** with the `playwright-test` MCP: call `generator_setup_page` with
93
- `project=<profile-id>` (for example `example-staging`) and `seedFile=tests/seed.spec.ts`, then perform the
94
- steps in the environment you were given (never ask for it; there is no default) to learn the locators.
95
- Do the read-only steps as written. **Never perform a step that writes or needs a choice**: the agent does not write
96
- data, in staging or in production, and it does not hand off to a person here or wait for one. Write those steps
97
- of the spec from the `## Inputs` recipe (positions, `[asrun]` names, control labels and roles that
98
- `/explore-case` recorded). Pages that only exist after such a step cannot be seen from here: write their locators
99
- from what the recipe and the exploration notes say, mark them in the report as "not verified on the page", and
100
- leave the first run of the spec (or `/heal-spec`) to confirm them. Do this for at least one profile
101
- per domain (example.com and exampleshop.com differ in markup). Profiles that are not logged in are skipped and
102
- named in the report. Use this exploration to answer the open points of step 2, and look at every profile that has
103
- an entry in `## Values` to read its value.
104
- Keep every browser call short (one action, about 60 s at most, a snapshot between steps, `{timeout: 5000}` when
105
- something may not appear, no third retry): see "Keep every browser call short" in `../exploring-cases/SKILL.md`.
106
- **Ready before you read, here too** (`readiness.md`): `browser_network_clear`, act, wait for the key requests
107
- (`browser_network_requests`) and the content signal, then snapshot. The request URL patterns for `actAndWait` and the
108
- `Ready when` signals of the case are taken from what you see here.
109
- **Check every target on the page:** for each target card, find its `Identified by` locator, confirm that it matches
110
- exactly one element and that this element is what `What` says (highlight it and read the screenshot). A locator that
111
- matches several elements or the wrong one is fixed or becomes a question, never `.first()`.
112
- Prefer `getByTestId` (the project uses `data-tn`), then role/label locators. No CSS chains or XPath.
113
- Update the case file with what you learned: fold observations into the right section marked
114
- `(observed, 🔍 to be confirmed by QA)`, remove the TODO blocks they answer (rewrite the sentence above the block),
115
- and narrow the ones that remain. Keep the shape of `open-questions.md` for every block. Edit the case
116
- with targeted edits; never touch its `## Source` table.
117
- 5. **Complete the case's `## Values`**: for every profile the spec runs for, add the entry with the literals the spec
118
- asserts (the same keys as `default`, only what differs). Do not move values into `profiles.yaml`; it holds connection
119
- settings only. Mention every value you added or corrected in the report.
120
- 6. **Write `tests/<id>.spec.ts`** (inside the group folder if the id has one, creating it if needed) following the conventions. One `test()` per outcome, and every
121
- case step wrapped in `test.step()` with the case's step text as the title, and the top-level
122
- `test.describe` tagged with `caseTags('<id>')`. The spec must
123
- contain no profile-specific literals: everything profile-specific comes from `requireValues('<id>', profile)`
124
- (`@asrunjs/asrun/values`), which merges `default` with the running profile's entry.
125
- Every action that loads something goes through `actAndWait` (`@asrunjs/asrun/ready`) with the key requests and the ready
126
- signal; every download goes through `verifyDownload` (`@asrunjs/asrun/downloads`); every check on a target has
127
- `// target: T<n>` on the line before it; every absence check has a `// ready: <signal>` before it. Never
128
- `waitForTimeout`, `networkidle` or `.first()` to pick an image. A confirmation or modal that may or may not appear is
129
- never decided by one `isVisible()` look (its `timeout` option is ignored): use `clickOptionalUntilHidden`
130
- (`@asrunjs/asrun/ready`) or a waiting `expect(...).toBeVisible({timeout})`; see `readiness.md`, "Optional confirmations".
131
- 7. **Check the files**: run `pnpm typecheck`, `pnpm spec:lint <id>` and `pnpm case:check` (or `node --import tsx <framework>/scripts/check-cases.ts` if
132
- it fails with a sandbox error such as `listen EPERM`). Fix every error, including every `FAIL` of the spec lint (a lint failure is never silenced: fix the spec or turn the check
133
- into a question). Every lint `WARN` is handled too: fix it, or name it in the report with the reason it is acceptable.
134
- Then do the **self-review**: for each assertion answer in writing "if the product were broken, would this fail?"
135
- and "if the product is right, could this fail by timing?" (a check that cannot fail, or that races the page, is rewritten).
136
- For every branch that depends on something possibly not there yet (`if`, `.count()`, `.catch(() => false)`), ask "what
137
- if it shows up a second later?"; the answer must be a retry loop or a waiting assertion, never a single look.
138
- Do NOT run `pnpm asrun`, do not use
139
- `--update-snapshots` and do not create baselines: the first run of the spec does that, and a person reviews them.
140
- 8. **Report** in the user's language: spec path, **an assertion table** (step, what is asserted, target card, ready signal, and
141
- `observed on the page` or `not verified`; no row may be missing), profiles explored, profiles skipped (and why), whether the case file was
142
- translated, what was added to the case file as observed, checks generated from observations (QA must confirm
143
- them), visual checks that need a baseline on the first run, and the remaining TODO blocks (`[blocking]` ones with
144
- the sentence that deleting the block would confirm). State that the spec was not run and give the command to run it:
145
- `pnpm asrun --env <env> --profile <ids> --grep <id>` (with `--update-snapshots` once if the spec has visual
146
- checks, then run `pnpm baselines:review <id> --new` and open every new image: it must show what its target card says). The last line is `RESULT: ready` (no `[blocking]` question left, spec
147
- written and the checks above pass) or `RESULT: needs-input`, as defined in `open-questions.md`.
148
-
149
- ## Cases that write data
150
- Only when the case front matter lists environments under `writes` (for example `writes: [staging]`). Never add
151
- `--allow-writes` or set `ASRUN_ALLOW_WRITES`. Use the `records` fixture, `configureCase` and `emailStepAllowed` as
152
- described in `references/conventions.md`. The agent does not perform write steps while generating (see step 4); the
153
- spec does, when a person runs it in an environment that `writes` lists. Elsewhere the runner skips the case with a
154
- notice. Never send email in production; in other environments only if `profile.email` is true, otherwise skip that step and say so
155
- in the report.
156
-
157
- ## Healing a failing spec
158
- Inputs: the case id and the environment (both required, there is no default: without an environment stop and say so),
159
- optionally `--profiles a,b`, `--report <run-id>` and `--hints <file>` (or free text after the arguments). Healing is
160
- the one case where this skill runs specs, and only in the environment given.
161
-
162
- Heal from evidence, not from a habit of rerunning. A report that already exists says what failed.
163
-
164
- 1. **Read the user's clues first** (the `--hints` file, or the free text). They say what to look at and may hold
165
- evidence the report lacks. They guide the diagnosis and nothing else: they never lower the bar for a product
166
- finding and never allow an expectation to be changed just to make a test pass. Quote the clue you used, or say that
167
- it did not hold up, in the report.
168
- 2. **Read the existing report**: run `pnpm heal:evidence --case <case-id> --env <env>` (add `--profile a,b` when `--profiles`
169
- was given). It reads `reports/*/summary.json` and prints, per profile, the status, the failed step, the error, the call
170
- log, the screenshots and videos, and ends with `EVIDENCE: fresh | stale | no-failure | none`. With `--report <run-id>`,
171
- read `reports/<run-id>/summary.json` instead. Look at the screenshot when the error alone does not explain it. A profile
172
- that failed only because it was not logged in is no evidence about the spec.
173
- 3. **Decide whether to rerun the case**. Do not rerun when the report is `fresh`, names the failing step and the error,
174
- and that is enough to tell whether the product changed or the test is wrong. Rerun when:
175
- - there is no report (`none`), or it is `stale` (the spec, the case or `profiles.yaml` changed after it);
176
- - the report does not explain the failure (a bare timeout with no locator, a failure before any step, a truncated log);
177
- - a clue from the user has to be checked against the real page;
178
- - a profile that must be healed has no result in the report (for example it was not logged in then).
179
- Rerun with one plain command, as few profiles as possible, and say why in the report:
180
- `pnpm asrun --env <env> --logged-in --profile <ids> --grep <case-id>` (optionally piped to `tail`; do not chain it
181
- with other commands, or it runs inside Claude's sandbox and fails with `listen EPERM`). Always add `--logged-in`: it
182
- leaves out the profiles that are not logged in instead of failing them. A profile that is not logged in is never healed:
183
- name it in the report with `pnpm signin <profile-id>`.
184
- 4. **Audit the assertion before you blame anyone.** Read the `assertion-targets.md` and `readiness.md` sections on healing.
185
- - Is the locator still what the target card says (`What`, `Identified by`)? Open the failure screenshot and the
186
- baseline (`pnpm baselines:review <case-id>`) and look at each: a baseline that shows something other than the card
187
- (a content image instead of a logo) is a wrong test, and passing runs on it prove nothing. Delete it and report it; a
188
- person creates the new baseline after the spec is fixed. Note: other profiles that pass on the same wrong target are also suspect.
189
- - Is the failure a "missing / empty / not visible / timeout" kind? Then it is first a readiness suspect: was the page
190
- still loading in the screenshot or video? Fix with `actAndWait` and a ready signal, never with a longer timeout alone or a sleep.
191
- - Is the spec weak in the ways the lint knows? Run `pnpm spec:lint <case-id>` and fix every `FAIL` while you are there.
192
- - A download that fails says which stage failed (start, finish, exists, valid): fix the stage, do not loosen the check.
193
- - Doubt about which element is meant: write the question into the case (a `[blocking]` TODO naming the exact image,
194
- text, place or UI) instead of choosing; do not rewrite the card quietly.
195
- 5. **Decide whether the PRODUCT changed or the TEST is wrong.**
196
- - Locator/timing problem: fix the spec.
197
- - A real product difference (wrong logo, wrong name): do NOT change the expectation to make it
198
- pass. Report it as a finding with the trace/screenshot path.
199
- 6. **After editing the spec, run it once to confirm** (same command as in step 3, for the profiles you healed). That rerun
200
- is the only one that is always allowed. Run `pnpm typecheck` too.
201
- 7. **Report** in the user's language: which report was read (or that there was none) and whether it was fresh or stale,
202
- whether the case was run again and why (or why not), what was changed, what is a product finding (with paths),
203
- which assertions were wrong or raced the page, which baselines are wrong and were deleted, what the user's clues contributed, which profiles were left out because they are not logged in, and the result of the
204
- confirming run.
205
-
206
- A case that writes data in `production` needs a person's confirmation at run time, so the rerun is refused there:
207
- use the report, or heal in staging, and never add `--allow-writes`.
1
+ ---
2
+ name: generating-specs
3
+ description: Turns a human-written test case (cases/<id>.md, the id may include a group folder such as documents/landing-page) into a Playwright spec (tests/<id>.spec.ts) for the multi-profile test suite, explores the real pages to learn the locators and values, and completes the case's own "## Values" block. It only generates: it never runs the tests. Use when asked to generate, update, fix or heal a spec, or for /gen-spec.
4
+ ---
5
+
6
+ # Generating specs from case documents
7
+
8
+ Read `references/conventions.md` first. It defines the spec shape, fixtures, assertion layers and
9
+ the read-only rules. Read the workspace `AGENTS.md` for the hard rules and the project rules. Read `../exploring-cases/references/open-questions.md`: it
10
+ defines how open questions are written into the case file, resolved and reported. Also read
11
+ `../exploring-cases/references/assertion-targets.md` (the exact thing each check looks at) and
12
+ `../exploring-cases/references/readiness.md` (wait for the content, verify downloads in five parts). All three are binding here.
13
+
14
+ This skill never stops to ask the user in the chat, because it may run without anybody present (a one-shot
15
+ `claude -p` / `opencode run`). Questions go into the case file; the final line is `RESULT: ready` or
16
+ `RESULT: needs-input`.
17
+
18
+ This skill trusts the case. What the pages show, what a person has to enter, and what the open questions are is
19
+ settled by `/explore-case` and recorded in the case (`(observed, ...)` marks, the step recipes). Generating
20
+ only finds locators and writes code: it replays the case and does not judge the case again.
21
+
22
+ Generating a spec and running it are separate processes. This skill only writes files: it does not run
23
+ `pnpm asrun`, does not create screenshot baselines and does not judge whether the spec passes. The user runs the
24
+ spec afterwards (see the report). Healing a failing spec (below) is the exception: it reads the last report first and runs only the profiles that are logged in, and only when the report is not enough.
25
+
26
+ ## Before you start
27
+ Read the workspace `AGENTS.md`: its §2 holds the project rules, and it points at the framework baseline. If
28
+ `.claude/skills/generating-specs.md` exists in the workspace, read it and apply it after this skill. Then apply
29
+ `AGENTS.md` §2, which wins over both (the baseline, "Precedence"). F1–F6 always win. This is not a stop condition:
30
+ continue with the procedure below.
31
+
32
+ ## Inputs
33
+ - The case id and the environment to explore in (`staging` or `production`). There is no default. If the
34
+ environment is missing, stop before exploring and report `RESULT: needs-input` ("pass the environment").
35
+ - Optionally `--force`: regenerate the spec from scratch and overwrite the existing one. See "An existing spec".
36
+
37
+ A case id is its path below `cases/` without `.md`: `hub-landing`, or `documents/landing-page` when the case is in a
38
+ folder (one level only). The spec is at the same path below `tests/` (`tests/documents/landing-page.spec.ts`); both import
39
+ the framework through the `@asrunjs/asrun/*` alias, so the folder depth does not change the imports.
40
+
41
+ - `cases/<id>.md` (front matter: `id`, `login: required|none` (`none` = public page; the login page itself is not a valid case), `suites: [<name>, ...]` (the old `priority: <level>` means one suite), `writes: [<env>, ...]` (omitted = read-only; the environments the spec may write in, e.g. `[staging]`) + `writes-note`, optional `envs: [...]`; sections Steps (with a step's `- Recipe:` lines) / Expected / Values / Visual checks / Targets; an old `## Inputs` section may still hold recipes). `## Values` holds one fenced yaml block keyed by `default` and by profile id, with the exact literals the spec asserts. The case document is the only place where expected values live.
42
+ - `profiles.yaml` (profiles: id, name, env, baseURL, account, tags, email). It holds no expected values.
43
+
44
+ ## An existing spec
45
+ Decide this first, before exploring anything. The spec path is `tests/<id>.spec.ts`.
46
+ - **No spec yet:** generate it (the procedure below).
47
+ - **A spec exists and `--force` was not given:** do not touch it. Do not edit the spec, the case values or any baseline.
48
+ Report that the spec already exists, say that `--force` regenerates and overwrites it, and give the id and
49
+ environment to run it again with. End with `RESULT: ready` (the spec that exists is not in question). Nothing else
50
+ is written, not even the case file.
51
+ - **A spec exists and `--force` was given:** the user chose to overwrite it. Regenerate it from the case with the
52
+ procedure below, and write the new file over the old one (do not merge with it: the case file is the source of
53
+ truth). Do not read the old spec as a starting point and do not keep it just because the result looks the same:
54
+ write the file from the case and the exploration. If it ends up identical, say so in the report. Keep the `## Values`
55
+ of other cases untouched; only this case's `## Values` is yours. Leave the
56
+ baseline images in `tests/<id>.spec.ts-snapshots/` alone: a changed visual check needs a person to update them,
57
+ so list in the report which baselines may now be out of date. Say in the report that an existing spec was
58
+ overwritten.
59
+
60
+ ## Procedure
61
+ 1. **Make sure the case uses the artifact language.** `AGENTS.md` §2 sets it (English by default), and spec step
62
+ titles copy the case's step text. If `cases/<id>.md` is written in another language, translate it into that
63
+ language and rewrite the file BEFORE anything else: keep the front matter keys and values, the section headings,
64
+ the `## Values` entries and the step numbering as they are, and keep quoted UI text exactly as it appears
65
+ on the page (do not translate strings the page shows). Do not add, drop or reinterpret any step or
66
+ expectation; if a sentence is unclear, translate it literally and raise it as a question in the next step.
67
+ Tell the user in the report that the case file was translated. Skip this when the case already uses that language.
68
+ 2. **Validate the case.** Collect everything that is ambiguous, every check without an expected value and every
69
+ missing per-profile value, including the `TODO` items already in the case. Resolve them in the order of
70
+ `open-questions.md`; do not stop here:
71
+ - **Steps that write or need a choice (`[input]`, `[writes]`) need a recipe.** Each must have at least one
72
+ `- Recipe (phase): ...` line under it in the case (written by `/explore-case`). An old case may still hold the
73
+ same recipe in `## Inputs`: use that line until it is moved under the step. If one has none, the case was
74
+ never explored with a person: stop before opening a browser, do not write the spec, and report `RESULT:
75
+ needs-input` with exactly this: "step <n> has no recorded input; run `/explore-case <id> <env> --attended`
76
+ first". Nothing else is written.
77
+ - Facts the page can answer and that the case does not hold yet are read in step 4 and written into the case as
78
+ `(observed, 🔍 to be confirmed by QA)`. A spec is generated from observed values.
79
+ - Safe defaults are applied: the suggestion is the sentence above the TODO block, and the block stays in the case
80
+ as `❓ TODO [non-blocking]` with an empty `✍️ Answer:`. A sentence with no TODO block under it is confirmed text.
81
+ - A non-empty `Answer:` is the user's answer: it wins over the sentence above it. Fold it into the case.
82
+ - Only what is always-blocking in `open-questions.md` blocks. Generate everything else; for a blocked check
83
+ (a `[blocking]` block), write `test.fixme(true, '<the question>')` inside its step (or skip that profile), never
84
+ an invented value and never a check built from the sentence above a `[blocking]` block.
85
+ - **Every check on a logo, image, icon, file, PDF, screenshot or repeated text needs a target card** (`## Targets`,
86
+ referenced as `[T<n>]`) that was observed (`Reference <profile-id>:` is a screenshot, not `none`) and has no open
87
+ `[blocking]` block. A check that has no card, or whose card is not observed, is not guessed here. Do not pick
88
+ an element yourself: report `RESULT: needs-input` with "check <text> has no observed target; run `/explore-case
89
+ <id> <env>` first" (or, when the card has a `[blocking]` TODO, write `test.fixme(true, '<the question>')` for that check as
90
+ the rule above says and go on with the rest).
91
+ - **Evidence gate:** a hard assertion (one that decides pass or fail) on something nobody has seen on a page is not
92
+ written. "Not verified on the page" is allowed for a locator of a page that only exists after a write step, but
93
+ never for what the check asserts about it: that goes to `test.fixme` with the reason, or `needs-input`.
94
+ - Do not invent expected values. If the case's `## Values` and the spec disagree, the case wins: fix the spec and
95
+ mention the change in the report.
96
+ 3. **Check sessions** if `login: required`: run `pnpm status --env <env>`. If a profile needs login, tell the
97
+ user to run `pnpm signin <profile-id>` and continue with the profiles that are ready. If `pnpm status`
98
+ cannot run (a sandbox error such as `listen EPERM` or a Chromium start failure), judge by the page instead: a page
99
+ that lands on the login page means the profile is not logged in.
100
+ 4. **Replay the case on the real page** with the `playwright-test` MCP. If its tools are not registered in this
101
+ session (no `browser_*` / `generator_*` tool) or it reports that it failed to start, do not fall back to ad-hoc
102
+ Playwright scripts: stop, tell the user which MCP server is missing and to run `pnpm asrun doctor`, write no spec,
103
+ and end with `RESULT: needs-input`.
104
+ Call `generator_setup_page` with
105
+ `project=<profile-id>` (for example `example-staging`) and `seedFile=tests/seed.spec.ts`, then perform the
106
+ steps in the environment you were given (never ask for it; there is no default) to learn the locators.
107
+ Do the read-only steps as written. **Never perform a step that writes or needs a choice**: the agent does not write
108
+ data, in staging or in production, and it does not hand off to a person here or wait for one. Write those steps
109
+ of the spec from the `- Recipe:` lines under the step (positions, `[asrun]` names, control labels and roles that
110
+ `/explore-case` recorded). Pages that only exist after such a step cannot be seen from here: write their locators
111
+ from what the recipe and the exploration notes say, mark them in the report as "not verified on the page", and
112
+ leave the first run of the spec (or `/heal-spec`) to confirm them. Do this for at least one profile
113
+ per domain (example.com and exampleshop.com differ in markup). Profiles that are not logged in are skipped and
114
+ named in the report. Use this exploration to answer the open points of step 2, and look at every profile that has
115
+ an entry in `## Values` to read its value.
116
+ Keep every browser call short (one action, about 60 s at most, a snapshot between steps, `{timeout: 5000}` when
117
+ something may not appear, no third retry): see "Keep every browser call short" in `../exploring-cases/SKILL.md`.
118
+ **Ready before you read, here too** (`readiness.md`): `browser_network_clear`, act, wait for the key requests
119
+ (`browser_network_requests`) and the content signal, then snapshot. The request URL patterns for `actAndWait` and the
120
+ `Ready when` signals of the case are taken from what you see here.
121
+ **Check every target on the page:** for each target card, find its `Identified by` locator, confirm that it matches
122
+ exactly one element and that this element is what `What` says (highlight it and read the screenshot). A locator that
123
+ matches several elements or the wrong one is fixed or becomes a question, never `.first()`.
124
+ Prefer `getByTestId` (the project uses `data-tn`), then role/label locators. No CSS chains or XPath.
125
+ Update the case file with what you learned: fold observations into the right section marked
126
+ `(observed, 🔍 to be confirmed by QA)`, remove the TODO blocks they answer (rewrite the sentence above the block),
127
+ and narrow the ones that remain. Keep the shape of `open-questions.md` for every block. Edit the case
128
+ with targeted edits; never touch its `## Source` table.
129
+ 5. **Complete the case's `## Values`**: for every profile the spec runs for, add the entry with the literals the spec
130
+ asserts (the same keys as `default`, only what differs). Do not move values into `profiles.yaml`; it holds connection
131
+ settings only. Mention every value you added or corrected in the report.
132
+ 6. **Write `tests/<id>.spec.ts`** (inside the group folder if the id has one, creating it if needed) following the conventions. One `test()` per outcome, and every
133
+ case step wrapped in `test.step()` with the case's step text as the title, and the top-level
134
+ `test.describe` tagged with `caseTags('<id>')`. The spec must
135
+ contain no profile-specific literals: everything profile-specific comes from `requireValues('<id>', profile)`
136
+ (`@asrunjs/asrun/values`), which merges `default` with the running profile's entry.
137
+ Every action that loads something goes through `actAndWait` (`@asrunjs/asrun/ready`) with the key requests and the ready
138
+ signal; every download goes through `verifyDownload` (`@asrunjs/asrun/downloads`); every check on a target has
139
+ `// target: T<n>` on the line before it; every absence check has a `// ready: <signal>` before it. Never
140
+ `waitForTimeout`, `networkidle` or `.first()` to pick an image. A confirmation or modal that may or may not appear is
141
+ never decided by one `isVisible()` look (its `timeout` option is ignored): use `clickOptionalUntilHidden`
142
+ (`@asrunjs/asrun/ready`) or a waiting `expect(...).toBeVisible({timeout})`; see `readiness.md`, "Optional confirmations".
143
+ 7. **Check the files**: run `pnpm typecheck`, `pnpm spec:lint <id>` and `pnpm case:check` (or `node --import tsx <framework>/scripts/check-cases.ts` if
144
+ it fails with a sandbox error such as `listen EPERM`). Fix every error, including every `FAIL` of the spec lint (a lint failure is never silenced: fix the spec or turn the check
145
+ into a question). Every lint `WARN` is handled too: fix it, or name it in the report with the reason it is acceptable.
146
+ Then do the **self-review**: for each assertion answer in writing "if the product were broken, would this fail?"
147
+ and "if the product is right, could this fail by timing?" (a check that cannot fail, or that races the page, is rewritten).
148
+ For every branch that depends on something possibly not there yet (`if`, `.count()`, `.catch(() => false)`), ask "what
149
+ if it shows up a second later?"; the answer must be a retry loop or a waiting assertion, never a single look.
150
+ Do NOT run `pnpm asrun`, do not use
151
+ `--update-snapshots` and do not create baselines: the first run of the spec does that, and a person reviews them.
152
+ 8. **Report** in the user's language: spec path, **an assertion table** (step, what is asserted, target card, ready signal, and
153
+ `observed on the page` or `not verified`; no row may be missing), profiles explored, profiles skipped (and why), whether the case file was
154
+ translated, what was added to the case file as observed, checks generated from observations (QA must confirm
155
+ them), visual checks that need a baseline on the first run, and the remaining TODO blocks (`[blocking]` ones with
156
+ the sentence that deleting the block would confirm). State that the spec was not run and give the command to run it:
157
+ `pnpm asrun --env <env> --profile <ids> --grep <id>` (with `--update-snapshots` once if the spec has visual
158
+ checks, then run `pnpm baselines:review <id> --new` and open every new image: it must show what its target card says). The last line is `RESULT: ready` (no `[blocking]` question left, spec
159
+ written and the checks above pass) or `RESULT: needs-input`, as defined in `open-questions.md`.
160
+
161
+ ## Cases that write data
162
+ Only when the case front matter lists environments under `writes` (for example `writes: [staging]`). Never add
163
+ `--allow-writes` or set `ASRUN_ALLOW_WRITES`. Use the `records` fixture, `configureCase` and `emailStepAllowed` as
164
+ described in `references/conventions.md`. The agent does not perform write steps while generating (see step 4); the
165
+ spec does, when a person runs it in an environment that `writes` lists. Elsewhere the runner skips the case with a
166
+ notice. Never send email in production; in other environments only if `profile.email` is true, otherwise skip that step and say so
167
+ in the report.
168
+
169
+ ## Healing a failing spec
170
+ Inputs: the case id and the environment (both required, there is no default: without an environment stop and say so),
171
+ optionally `--profiles a,b`, `--report <run-id>` and `--hints <file>` (or free text after the arguments). Healing is
172
+ the one case where this skill runs specs, and only in the environment given.
173
+
174
+ Heal from evidence, not from a habit of rerunning. A report that already exists says what failed.
175
+
176
+ 1. **Read the user's clues first** (the `--hints` file, or the free text). They say what to look at and may hold
177
+ evidence the report lacks. They guide the diagnosis and nothing else: they never lower the bar for a product
178
+ finding and never allow an expectation to be changed just to make a test pass. Quote the clue you used, or say that
179
+ it did not hold up, in the report.
180
+ 2. **Read the existing report**: run `pnpm heal:evidence --case <case-id> --env <env>` (add `--profile a,b` when `--profiles`
181
+ was given). It reads `reports/*/summary.json` and prints, per profile, the status, the failed step, the error, the call
182
+ log, the screenshots and videos, and ends with `EVIDENCE: fresh | stale | no-failure | none`. With `--report <run-id>`,
183
+ read `reports/<run-id>/summary.json` instead. Look at the screenshot when the error alone does not explain it. A profile
184
+ that failed only because it was not logged in is no evidence about the spec.
185
+ 3. **Decide whether to rerun the case**. Do not rerun when the report is `fresh`, names the failing step and the error,
186
+ and that is enough to tell whether the product changed or the test is wrong. Rerun when:
187
+ - there is no report (`none`), or it is `stale` (the spec, the case or `profiles.yaml` changed after it);
188
+ - the report does not explain the failure (a bare timeout with no locator, a failure before any step, a truncated log);
189
+ - a clue from the user has to be checked against the real page;
190
+ - a profile that must be healed has no result in the report (for example it was not logged in then).
191
+ Rerun with one plain command, as few profiles as possible, and say why in the report:
192
+ `pnpm asrun --env <env> --logged-in --profile <ids> --grep <case-id>` (optionally piped to `tail`; do not chain it
193
+ with other commands, or it runs inside Claude's sandbox and fails with `listen EPERM`). Always add `--logged-in`: it
194
+ leaves out the profiles that are not logged in instead of failing them. A profile that is not logged in is never healed:
195
+ name it in the report with `pnpm signin <profile-id>`.
196
+ 4. **Audit the assertion before you blame anyone.** Read the `assertion-targets.md` and `readiness.md` sections on healing.
197
+ - Is the locator still what the target card says (`What`, `Identified by`)? Open the failure screenshot and the
198
+ baseline (`pnpm baselines:review <case-id>`) and look at each: a baseline that shows something other than the card
199
+ (a content image instead of a logo) is a wrong test, and passing runs on it prove nothing. Delete it and report it; a
200
+ person creates the new baseline after the spec is fixed. Note: other profiles that pass on the same wrong target are also suspect.
201
+ - Is the failure a "missing / empty / not visible / timeout" kind? Then it is first a readiness suspect: was the page
202
+ still loading in the screenshot or video? Fix with `actAndWait` and a ready signal, never with a longer timeout alone or a sleep.
203
+ - Is the spec weak in the ways the lint knows? Run `pnpm spec:lint <case-id>` and fix every `FAIL` while you are there.
204
+ - A download that fails says which stage failed (start, finish, exists, valid): fix the stage, do not loosen the check.
205
+ - Doubt about which element is meant: write the question into the case (a `[blocking]` TODO naming the exact image,
206
+ text, place or UI) instead of choosing; do not rewrite the card quietly.
207
+ 5. **Decide whether the PRODUCT changed or the TEST is wrong.**
208
+ - Locator/timing problem: fix the spec.
209
+ - A real product difference (wrong logo, wrong name): do NOT change the expectation to make it
210
+ pass. Report it as a finding with the trace/screenshot path.
211
+ 6. **After editing the spec, run it once to confirm** (same command as in step 3, for the profiles you healed). That rerun
212
+ is the only one that is always allowed. Run `pnpm typecheck` too.
213
+ 7. **Report** in the user's language: which report was read (or that there was none) and whether it was fresh or stale,
214
+ whether the case was run again and why (or why not), what was changed, what is a product finding (with paths),
215
+ which assertions were wrong or raced the page, which baselines are wrong and were deleted, what the user's clues contributed, which profiles were left out because they are not logged in, and the result of the
216
+ confirming run.
217
+
218
+ A case that writes data in `production` needs a person's confirmation at run time, so the rerun is refused there:
219
+ use the report, or heal in staging, and never add `--allow-writes`.