@asrunjs/asrun 0.2.0 → 0.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/explore-case.md +18 -15
- package/.claude/commands/gen-spec.md +16 -13
- package/.claude/commands/heal-spec.md +26 -23
- package/.claude/commands/make-case.md +11 -8
- package/.claude/skills/exploring-cases/SKILL.md +146 -132
- package/.claude/skills/exploring-cases/references/open-questions.md +116 -112
- package/.claude/skills/exploring-cases/references/readiness.md +146 -137
- package/.claude/skills/exploring-cases/references/user-handoff.md +157 -147
- package/.claude/skills/generating-specs/SKILL.md +219 -207
- package/.claude/skills/generating-specs/references/conventions.md +148 -148
- package/.claude/skills/making-cases/SKILL.md +72 -64
- package/.claude/skills/making-cases/references/normalizing.md +39 -38
- package/.claude/skills/making-cases/references/writes-and-email.md +37 -36
- package/AGENTS.md +159 -154
- package/CLAUDE.md +2 -1
- package/README.md +512 -410
- package/dist/lib/build-artifacts.d.ts +8 -0
- package/dist/lib/build-artifacts.js +10 -0
- package/dist/lib/case-doc.d.ts +311 -0
- package/dist/lib/case-doc.js +1117 -0
- package/dist/lib/cases.d.ts +2 -15
- package/dist/lib/cases.js +5 -78
- package/dist/lib/command.d.ts +24 -0
- package/dist/lib/command.js +90 -0
- package/dist/lib/fixtures.js +2 -2
- package/dist/lib/login-card.d.ts +22 -0
- package/dist/lib/login-card.js +183 -0
- package/dist/lib/node-cli.d.ts +6 -2
- package/dist/lib/node-cli.js +16 -6
- package/dist/lib/profiles.d.ts +25 -1
- package/dist/lib/profiles.js +54 -1
- package/dist/lib/run-filters.js +10 -2
- package/dist/lib/session.d.ts +13 -1
- package/dist/lib/session.js +73 -7
- package/dist/lib/targets.d.ts +5 -13
- package/dist/lib/targets.js +4 -29
- package/dist/lib/values.js +2 -2
- package/dist/scripts/build.js +2 -1
- package/dist/scripts/check-cases.js +74 -6
- package/dist/scripts/doctor.js +115 -4
- package/dist/scripts/explore.js +22 -1
- package/dist/scripts/init.d.ts +7 -1
- package/dist/scripts/init.js +61 -13
- package/dist/scripts/lint-specs.js +4 -1
- package/dist/scripts/login.js +30 -2
- package/dist/ui/server/actions.d.ts +4 -0
- package/dist/ui/server/actions.js +25 -12
- package/dist/ui/server/ai-models.d.ts +3 -0
- package/dist/ui/server/ai-models.js +347 -0
- package/dist/ui/server/ai-stream.d.ts +34 -9
- package/dist/ui/server/ai-stream.js +283 -77
- package/dist/ui/server/app.js +74 -12
- package/dist/ui/server/jobs.d.ts +7 -1
- package/dist/ui/server/jobs.js +69 -15
- package/dist/ui/server/profiles.d.ts +20 -1
- package/dist/ui/server/profiles.js +214 -38
- package/dist/ui/shared/ai-commands.d.ts +16 -6
- package/dist/ui/shared/ai-commands.js +35 -12
- package/dist/ui/shared/types.d.ts +114 -0
- package/docs/agents/framework-dev.md +79 -0
- package/docs/theme-guide.html +922 -692
- package/package.json +12 -1
- package/templates/.claude/commands/explore-case.md +22 -21
- package/templates/.claude/commands/gen-spec.md +21 -20
- package/templates/.claude/commands/heal-spec.md +31 -30
- package/templates/.claude/commands/make-case.md +13 -12
- package/templates/.mcp.json +8 -8
- package/templates/AGENTS.md +64 -45
- package/templates/cases/_template.md +57 -63
- package/templates/opencode.jsonc +16 -11
- package/templates/profiles.yaml +42 -29
- package/ui/dist/assets/{dist-BHaIOZdW.js → dist-BGbe0-Aa.js} +1 -1
- package/ui/dist/assets/{dist-CBV7_vDN.js → dist-Ba-HC662.js} +1 -1
- package/ui/dist/assets/{dist-BiE4suo9.js → dist-BtMLMthJ.js} +1 -1
- package/ui/dist/assets/{dist-gF2bFn-5.js → dist-C-Oyo1ZT.js} +1 -1
- package/ui/dist/assets/{dist-DX8knLVr.js → dist-C8MdhqgG.js} +1 -1
- package/ui/dist/assets/{dist-DHCY9KpV.js → dist-CG1pHn5m.js} +1 -1
- package/ui/dist/assets/{dist-BwYRbe5s.js → dist-CmpteMYY.js} +1 -1
- package/ui/dist/assets/{dist-D_I45xgJ.js → dist-DSy3dVf4.js} +1 -1
- package/ui/dist/assets/{dist-BqAROxzI.js → dist-Dq6sW5Ur.js} +1 -1
- package/ui/dist/assets/{dist-BSJMM_jp.js → dist-jjxQutEm.js} +1 -1
- package/ui/dist/assets/index-CTmnF0Wp.css +2 -0
- package/ui/dist/assets/index-P_Yo0ZZn.js +338 -0
- package/ui/dist/help/ai-tools.png +0 -0
- package/ui/dist/help/case-explore.png +0 -0
- package/ui/dist/help/case-generate.png +0 -0
- package/ui/dist/help/cases-memo.png +0 -0
- package/ui/dist/help/profile-form.png +0 -0
- package/ui/dist/help/profiles-login.png +0 -0
- package/ui/dist/help/reports.png +0 -0
- package/ui/dist/help/run-tests.png +0 -0
- package/ui/dist/index.html +22 -16
- package/ui/dist/assets/index-DmasQ6Rq.css +0 -2
- package/ui/dist/assets/index-I7uU93kk.js +0 -294
package/README.md
CHANGED
|
@@ -1,410 +1,512 @@
|
|
|
1
|
-
# AsRun
|
|
2
|
-
|
|
3
|
-
Automated tests against production and staging for multiple profiles. Humans write test cases in plain language; an AI
|
|
4
|
-
agent turns them into Playwright specs; the specs run locally, once per profile, and produce videos, screenshots and
|
|
5
|
-
traces for review.
|
|
6
|
-
|
|
7
|
-
Profiles and environments: see `profiles.yaml`. A profile is one environment of the system under test (a URL, an
|
|
8
|
-
account label, tags and an environment name), and each one becomes one Playwright project.
|
|
9
|
-
|
|
10
|
-
## Profiles and environments
|
|
11
|
-
Every entry in `profiles.yaml` is one environment: `id`, `name`, `env`, `baseURL`, optional `account
|
|
12
|
-
`email`. One profile = one Playwright project named after its
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
`
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
- **
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
the
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
pnpm
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
`
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
the
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
|
118
|
-
|
|
119
|
-
|
|
|
120
|
-
|
|
|
121
|
-
|
|
|
122
|
-
|
|
|
123
|
-
|
|
|
124
|
-
|
|
|
125
|
-
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
|
136
|
-
|
|
137
|
-
|
|
|
138
|
-
|
|
|
139
|
-
|
|
|
140
|
-
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
the
|
|
319
|
-
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
AGENTS.md
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
`
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
-
|
|
410
|
-
|
|
1
|
+
# AsRun
|
|
2
|
+
|
|
3
|
+
Automated tests against production and staging for multiple profiles. Humans write test cases in plain language; an AI
|
|
4
|
+
agent turns them into Playwright specs; the specs run locally, once per profile, and produce videos, screenshots and
|
|
5
|
+
traces for review.
|
|
6
|
+
|
|
7
|
+
Profiles and environments: see `profiles.yaml`. A profile is one environment of the system under test (a URL, an
|
|
8
|
+
account label, tags and an environment name), and each one becomes one Playwright project.
|
|
9
|
+
|
|
10
|
+
## Profiles and environments
|
|
11
|
+
Every entry in `profiles.yaml` is one environment: `id`, `name`, `env`, `baseURL`, optional `account`, `tags`,
|
|
12
|
+
`startPath`, `sessionCheckPath` and `sessionCheck`, and `email`. One profile = one Playwright project named after its
|
|
13
|
+
id (for example `example-staging`).
|
|
14
|
+
- **There is no default environment for a run.** `pnpm asrun` and every AI command need `--env <env>`, which selects
|
|
15
|
+
the profiles whose `env` matches (the Run tests page needs one radio button picked). `pnpm signin <profile-id>` and
|
|
16
|
+
`pnpm explore <profile-id>` need no `--env`: the profile already names its environment.
|
|
17
|
+
- **Separate logins.** Sessions are saved per profile in `.auth/<profile-id>.json`; log in with
|
|
18
|
+
`pnpm signin <profile-id>` (a human, including 2FA).
|
|
19
|
+
- **Separate baselines and reports.** Screenshot baselines are stored per project (`...-example-staging-darwin.png`).
|
|
20
|
+
A run's folder, report title and summary carry the environment (`reports/2026-10-28T07-30-22-staging`); the web UI
|
|
21
|
+
shows PRODUCTION in red and STAGING in blue everywhere, and prints `ENV: PRODUCTION` / `ENV: STAGING` at the start of a CLI run.
|
|
22
|
+
- **Tags.** `tags: [us, core]` groups profiles; `pnpm asrun --env staging --tag us` runs only the matching ones.
|
|
23
|
+
- **Email.** `email: true` says a real mailbox is connected and tests may send email. It defaults to `false`, is set to
|
|
24
|
+
`false` for new profiles, and is rejected for `production`.
|
|
25
|
+
- **Visit start path.** `startPath` (`/app/home`) is the page the Profiles page's "Visit" opens; empty opens the site
|
|
26
|
+
root. It must be a path on the site (a single leading `/`, no spaces), so it can never leave the site.
|
|
27
|
+
`pnpm explore <profile-id>` without a path still opens the root. `sessionCheckPath` (default `/account/`) is the
|
|
28
|
+
page a saved session is checked against, and `sessionCheck` says how that page shows a missing login: the
|
|
29
|
+
address a signed-out visitor is sent to (default `/login`), a selector that only exists while signed in, a
|
|
30
|
+
selector that proves the visitor is signed out, and how long to wait for one of them. Sites differ: a
|
|
31
|
+
single-page app redirects after it has started, some answer `401`, some show the login form at the same
|
|
32
|
+
address. Without a `loggedInSelector`, "not sent to the login page" counts as signed in, which can mislead on
|
|
33
|
+
a slow single-page app.
|
|
34
|
+
- **Expected values are not here.** They live in each case document (`## Values`); `profiles.yaml` holds connection
|
|
35
|
+
settings only, so adding a profile never touches a case and two profiles can never fight over one case's values.
|
|
36
|
+
|
|
37
|
+
## Installing asrun in a project
|
|
38
|
+
|
|
39
|
+
`asrun` is a normal devDependency: it brings the Playwright fixtures and config, the case/spec tooling, the agent
|
|
40
|
+
prompts and the web UI. A project commits only its own test content.
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
pnpm dlx @asrunjs/asrun init # scaffolding only: writes .env, configs, AGENTS.md, commands, the build-script approval
|
|
44
|
+
pnpm add -D @asrunjs/asrun # or: pnpm add -D github:wizcas/asrun#v0.1.0
|
|
45
|
+
pnpm exec asrun init --update # picks up the installed version
|
|
46
|
+
pnpm go # install + Chromium + doctor + web UI
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Running `init` before `add` matters on pnpm 12: it writes the `allowBuilds` entry that pnpm needs before it will run
|
|
50
|
+
the install scripts of `esbuild`/`ffmpeg-static` (without it, `pnpm add` exits with `ERR_PNPM_IGNORED_BUILDS`).
|
|
51
|
+
`pnpm dlx @asrunjs/asrun init --install` does both in one step (the spec is optional:
|
|
52
|
+
`--install github:wizcas/asrun#v0.1.0`).
|
|
53
|
+
|
|
54
|
+
`pnpm asrun init` is safe to run again: it never overwrites a file you own.
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
pnpm asrun init --update # refreshes only the generated files (playwright.config.ts, seed.spec.ts, commands)
|
|
58
|
+
pnpm asrun doctor # checks profiles, the tsconfig alias, generated files, the installed version, skill overrides
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
The project commits `profiles.yaml`, `cases/`, `tests/` (with their `*-snapshots/`), `AGENTS.md`, `.claude/`,
|
|
62
|
+
`.mcp.json`, `opencode.jsonc` and the generated files. It never commits `.auth/`, `reports/`, `.tmp/`, `.explore/`,
|
|
63
|
+
`.env` or `node_modules/`. See "Layout" below.
|
|
64
|
+
|
|
65
|
+
**Which workspace a command runs against** comes from `.env` (`ASRUN_ROOT`, relative to that file; gitignored and
|
|
66
|
+
written by `init`), then from the nearest folder with `profiles.yaml`/`.asrun.json`. A real environment variable wins
|
|
67
|
+
over `.env`, so a CI job can point a checkout somewhere else without editing files.
|
|
68
|
+
|
|
69
|
+
Update:
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
pnpm add -D @asrunjs/asrun@latest # or: pnpm add -D github:wizcas/asrun#v0.2.0
|
|
73
|
+
pnpm exec asrun init --update
|
|
74
|
+
pnpm asrun doctor
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
There is no folder to move: the framework lives in `node_modules/@asrunjs/asrun`, which the package manager owns. To change
|
|
78
|
+
the framework itself, `pnpm patch asrun` (a committed patch) or point the dependency at your fork's tag.
|
|
79
|
+
|
|
80
|
+
`templates/` in the package holds what `init` writes. `AGENTS.md` and the skills stay in the package as the single
|
|
81
|
+
source and carry no product-specific rules. A project states what differs in §2 of its own `AGENTS.md`: name the
|
|
82
|
+
rule id (`F1`..`F10`) and say whether it `narrows` it (stricter) or `replaces` it (a different default, for example
|
|
83
|
+
the artifact language); `F1`..`F6` cannot be changed. A project may also refine a skill's procedure with
|
|
84
|
+
`.claude/skills/<skill>.md`. `pnpm asrun doctor` checks the table and lists the overrides, and an update never
|
|
85
|
+
rewrites a project's `AGENTS.md`, `CLAUDE.md` or overrides (`init --force` refuses them without `--reset-rules`).
|
|
86
|
+
|
|
87
|
+
## Quick start in a project
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
node framework/scripts/go.mjs # the very first time (no dependencies yet)
|
|
91
|
+
pnpm go # every time after that
|
|
92
|
+
```
|
|
93
|
+
That is all. It checks Node (22+), lists the framework in `pnpm-workspace.yaml`, installs the dependencies and the
|
|
94
|
+
browser (`pnpm install` + Chromium), writes `.env` and the workspace files, runs `doctor`, starts the web UI and opens
|
|
95
|
+
http://127.0.0.1:4173. Run it again any day: the install step then takes seconds, and if the UI is already running it
|
|
96
|
+
just opens it. The **?** icon in the top bar of the UI opens the step-by-step "How to start" page (install AsRun in a
|
|
97
|
+
project, choose the AI tool, add a profile, log in, make cases, explore, generate the spec, run the tests, read the
|
|
98
|
+
report). `pnpm go --no-open` does not open the browser.
|
|
99
|
+
|
|
100
|
+
The tab title shows where you are (`AsRun | Cases | documents/landing-page`, `AsRun | Job Log`). When it is your
|
|
101
|
+
turn (a login to finish, a step that an attended exploration hands to you in the browser window, or answers that an
|
|
102
|
+
AI run needs), the title gets a count in front (`(1) AsRun | ...`), the job shows a "Your turn" badge, and the bell
|
|
103
|
+
icon in the top bar, once switched on, shows a browser notification: it comes when the tab goes to the background (or
|
|
104
|
+
at once, if it already was) and stays until you click or close it. A test run or an AI run that ends while the tab is
|
|
105
|
+
in the background is announced the same way. Clicking a notification opens that job.
|
|
106
|
+
|
|
107
|
+
You need Node 22+ and pnpm. The AI steps (make cases, explore, generate) also need Claude Code or OpenCode; `pnpm go`
|
|
108
|
+
warns if neither is found, everything else works without them.
|
|
109
|
+
`ffmpeg` is also needed: it joins the videos of one test (a new browser tab has its own recording) into a single video
|
|
110
|
+
for the report. `pnpm go` tries to install it (Homebrew on macOS, winget on Windows; on Linux run
|
|
111
|
+
`sudo apt install ffmpeg`). If it is missing, the optional `ffmpeg-static` package is used instead; if neither works,
|
|
112
|
+
the report shows the videos separately.
|
|
113
|
+
Then check `profiles.yaml`: the profile ids, names, environments and addresses.
|
|
114
|
+
|
|
115
|
+
## Every day
|
|
116
|
+
|
|
117
|
+
| I want to... | Command |
|
|
118
|
+
|---|---|
|
|
119
|
+
| Check the workspace (profiles, tsconfig alias, generated files, skill overrides) | `pnpm asrun doctor` |
|
|
120
|
+
| Create or refresh the generated workspace files | `pnpm asrun init`, `pnpm asrun init --update` |
|
|
121
|
+
| See which profiles are logged in | `pnpm status` (all environments) or `pnpm status --env staging` |
|
|
122
|
+
| Log in a profile (human, with 2FA) | `pnpm signin <profile-id>` |
|
|
123
|
+
| Run everything in one environment | `pnpm asrun --env staging` |
|
|
124
|
+
| Run one profile | `pnpm asrun --env production --profile example-production` |
|
|
125
|
+
| Run all profiles except some (e.g. not logged in) | `pnpm asrun --env production --skip-profile example-production,example-staging` |
|
|
126
|
+
| Run only the profiles that carry a tag | `pnpm asrun --env staging --tag us` |
|
|
127
|
+
| Run only the profiles that are logged in | `pnpm asrun --env production --logged-in` |
|
|
128
|
+
| Read what the last report says about a case | `pnpm heal:evidence --case <case-id> --env staging` |
|
|
129
|
+
| Check the specs for guessed targets, sleeps and weak download checks | `pnpm spec:lint [case-id]` |
|
|
130
|
+
| Render a PDF to images and print its text (to look at it) | `pnpm pdf:render <file.pdf> --out .explore/targets/<case-id>/<name> [--pages all] [--corner top-left]` |
|
|
131
|
+
| List screenshot baselines next to what each must show | `pnpm baselines:review [case-id] [--new]` |
|
|
132
|
+
| Overwrite the baselines of some cases | `pnpm asrun --env staging --grep <case-id> --update-baselines <case-id>` |
|
|
133
|
+
| Run one case | `pnpm asrun --env production --grep hub-landing` |
|
|
134
|
+
| Run cases of one suite | `pnpm asrun --env production --suite smoke` (several: `--suite smoke,regression`; `--priority` is the old name) |
|
|
135
|
+
| Leave out the cases that write data | `pnpm asrun --env production --read-only` |
|
|
136
|
+
| List test data a case could not remove | `pnpm cleanup --env <env>` (`--forget` after deleting it by hand) |
|
|
137
|
+
| Watch a run step by step | `pnpm asrun --env staging --ui` |
|
|
138
|
+
| Show the current test step as text on the videos | `pnpm asrun --env production --step-text` (also `=1\|true\|yes\|on`; off by default) |
|
|
139
|
+
| Open the last report (videos, screenshots, traces) | `pnpm report` |
|
|
140
|
+
| Look around a page as a profile (files you download in that window are saved to `~/Downloads`; the path is printed in the terminal / job log) | `pnpm explore <profile-id> /some/path/` |
|
|
141
|
+
|
|
142
|
+
### Local web UI
|
|
143
|
+
```bash
|
|
144
|
+
pnpm go # install + build + start + open the browser (use this the first time)
|
|
145
|
+
pnpm ui # builds the web app and serves http://127.0.0.1:4173
|
|
146
|
+
pnpm ui:dev # development: Vite on :5173 with hot reload, API on :4173
|
|
147
|
+
```
|
|
148
|
+
One page for the daily work (it only listens on `127.0.0.1`):
|
|
149
|
+
|
|
150
|
+
| Page | What it does |
|
|
151
|
+
|---|---|
|
|
152
|
+
| Profiles | Login state of every profile, one row each (same check as `pnpm status`). "Log in" starts `pnpm signin <profile-id>`; log in in the browser window that opens, then press "Done, save the session" on the card in that window's bottom-left corner (the "I've logged in, save the session" button on the job does the same, from the UI). A logged-in profile shows `Login YES`. Every row has a "..." menu: a logged-in profile gets "Verify" (same read-only session check) and "Re-login" (starts `pnpm signin` again), a saved session gets "Logout" (deletes the saved session file, after a confirmation), and every row gets "Duplicate" (adds a second entry with the same settings under a new id; the saved login is not copied) and "Delete" (removes the entry from `profiles.yaml` and deletes its saved login, after a confirmation; screenshot baselines and old reports keep the id in their names and stay on disk). "Visit" (shown for logged-in profiles) opens a logged-in browser window at the profile's `startPath`, or the site root when it is empty (`pnpm explore <profile-id> <startPath>`). While that window is open the button turns into a red "Close browser", which kills the browser process. Files downloaded in that window are saved to `~/Downloads` (an existing file is never overwritten, a ` (1)` is added) and the saved path is printed in the job log. "Add a new profile" has a form (environment, URL, account label, tags, start path) that appends an entry to `profiles.yaml`, plus a guide for what to do next (log in, update the cases, a copyable AI prompt). |
|
|
153
|
+
| Cases | List of `cases/**/*.md`, grouped by folder (collapsible), source editor with a rendered preview (both highlight what needs a human: `TODO` in any spelling such as `todo` / `TO-DO` / `to-do` (red for `🚫 TODO [blocking]`, gold otherwise), `🔍 to be confirmed by QA` / `👁️ not observed` / `TBC` / `TBD` in blue, and your `✍️ Answer:` lines in green; the emoji in front is part of the highlighted marker; hover a highlighted word for a short tooltip on what it means and how to resolve it, and click the small X after it to delete the marker, or the whole question block for a `TODO`, in one step, in the editor and in the preview (Undo brings it back; the file is only saved when you save). An `Answer:` gets its X once you typed something, and it clears what you typed), the editor and the preview scroll together, the preview block that holds the caret is shaded, and a click in the preview puts the caret on that line of the editor; `Ctrl/Cmd+S` to save, "New case" has an optional folder field and runs `pnpm case:new`. The **Spec** view is editable too (`Ctrl/Cmd+S` saves; the save is refused when the file changed on disk meanwhile): **Open in editor** opens the spec in the default application of your system for `.spec.ts` files, and **Copy path** copies its absolute path, to paste into a terminal editor such as nvim. Click the file name (or the pencil icon next to it) to rename the case or move it to another folder: the spec and the baseline images move with it and the spec is rewritten to match. **Delete** (type the id to confirm, or click the feather icon to fill it in) removes the case, its spec and its baseline images; both dialogs list exactly what they touch first, and are refused while an AI or test run is in progress. The editor toolbar has a command picker (explore / generate / heal, explore by default), an **Environment** picker (required for every command, no default), an **Attended** box (ticked by default, only for explore), a **Run AI** button (for **Heal spec** it opens a dialog first, see below) and icon buttons to copy the shell command and open the AI guide, for the one-shot AI command of the open case. The Claude / OpenCode switch (Claude by default, remembered in the browser) is in the top bar and applies to every AI command. The **Model** picker beside it chooses the model those commands run with, out of the models that CLI has on this machine (read from `opencode models`, and from Claude Code's own settings, gateway or aliases; searchable, remembered per CLI); without a choice, OpenCode's free built-in model is what runs. The **Thinking** icon beside the picker turns the agent's thinking on for every AI run and lights up while it is on (off by default). **From memo** opens a Markdown editor with its rendered preview beside it for raw test notes: **Make cases** saves them to `.tmp/memos/` and runs `/make-case` one-shot (no questions: defaults for gaps, a `TODO` block for unknown values, no page exploration, existing cases are never overwritten); **Copy /make-case** copies the memo for the full interactive dialogue. When the run ends, the **What to do next** bar shows how many cases are new; **Review** opens a sheet listing each one, with an **Explore** button that opens the case with Explore case selected: a case written from a memo has not been checked against the real pages yet. |
|
|
154
|
+
| Run tests | Form for `pnpm asrun`: environment (required, no default), profiles, suite, read-only, case, step text. The **Case** field is a searchable picker: type part of a case id and pick it from the list (only cases that have a spec are offered). The **Cases to run** list shows what the filters select, with an **Update baselines** box per case (`--update-baselines`, unticked by default; asks before it overwrites and shows the new images next to their target cards when the run ends). Asks for confirmation when write cases would run in production. |
|
|
155
|
+
| Reports | One entry per run, each with a PRODUCTION / STAGING badge (runs from before environments existed are shown as PRODUCTION). **Results** lists every case x profile with the failure reason on one line (failed first); selecting one shows a red "Why it failed" block (failed step + error message) above the screenshot, video and a folded "Technical details" (call log, stack trace). **Full Playwright report** embeds the standard HTML report (videos and traces work). The latest run opens first. Runs made before summaries existed only have the full report. |
|
|
156
|
+
| Help (the **?** icon in the top bar) | "How to start": nine short steps with screenshots, from installing AsRun to reading a report. The screenshots are in `ui/web/public/help/` (retake them when a page changes). |
|
|
157
|
+
| AI guide | How each AI step works in the Web UI (which runs the AI tool for you), and the same `/make-case`, `/explore-case`, `/gen-spec`, `/heal-spec` commands for Claude Code and OpenCode in a terminal. |
|
|
158
|
+
|
|
159
|
+
The case editor opens in **Single** with the **Panel** full width. The toolbar switches it to **Split** (source on the
|
|
160
|
+
left, one pane on the right; drag the divider to resize them, double-click it to reset), or, still in Single, to
|
|
161
|
+
**Editor** or **Doc** (the case rendered as a document). In Split the right pane is **Panel** (the default) or **Doc**;
|
|
162
|
+
its switch sits at the right end of the toolbar, because that is the pane it controls. **Panel**
|
|
163
|
+
shows the case as a form, with what needs a person first: the open questions (the text above each one is what the case
|
|
164
|
+
will say, so it is shown in full — accept it to delete the question, or write an answer and the AI folds it in; a click
|
|
165
|
+
anywhere else on a question jumps to it in the source editor and flashes the text above with the whole block, while the
|
|
166
|
+
answer field and the buttons keep their own behaviour) and the
|
|
167
|
+
`🔍 to be confirmed by QA` / `👁️ not observed` markers (click a marker to jump to its line in the source, or confirm it
|
|
168
|
+
to remove the marker and keep the sentence; the strip at the top counts them, so a collapsed list still says what
|
|
169
|
+
waits). Then the front matter (title, login, suites, writes and its note, environments), the steps (the sentence, its
|
|
170
|
+
`[writes]` / `[input]` / `[email]` tags, the `- Ready when:` lines — the signal that the content a step loads is really
|
|
171
|
+
there, which the generated test waits for before it looks — and the `- Recipe:` lines of the steps a person does: what
|
|
172
|
+
they did in the browser, recorded by `/explore-case --attended`; several lines per step are normal, and they are not
|
|
173
|
+
test data — the asserted values are under Values), the `## Expected` and `## Visual checks` bullets (drag the handle, or
|
|
174
|
+
Space and the arrow keys, to reorder them with their questions), the `## Values` table (one column per profile; an empty
|
|
175
|
+
cell inherits `default`, clearing a cell removes it from that profile), and the `## Targets` cards, read-only: one row
|
|
176
|
+
per card, with the fields it is missing and whether no check uses it. Every `[T1]` in a step, in a ready signal or next
|
|
177
|
+
to a check is a chip — orange, because it is a reference, with a white fill when the card is ready, amber when it needs
|
|
178
|
+
work and red when there is no card — and clicking it (or a Targets row) opens the card in a window: what it is, where it
|
|
179
|
+
is, how it is found, what it must not be confused with, and its reference screenshots per profile (click one for the
|
|
180
|
+
full size), with **Open in editor** to edit the text. The screenshots are local state under `.explore/` (git-ignored): on a machine
|
|
181
|
+
that never ran the exploration the card keeps the path and says the file is not there. The panel's text
|
|
182
|
+
fields (a step sentence, its ready signals and recipe lines, a bullet, an answer, the writes note) are small Markdown
|
|
183
|
+
editors with the source editor's highlighting and monospace font; they grow with their text and scroll past a height
|
|
184
|
+
limit. `## Goal`, `## Notes` and `## Source` are read-only: clicking a paragraph moves the caret of
|
|
185
|
+
the source editor to that line and flashes it three times; in **Single** a click there (and every **Open in editor**
|
|
186
|
+
button) also switches the pane back to **Editor**, so the line never opens in a pane nobody sees.
|
|
187
|
+
Every section collapses, each title carries the icon of what the section holds, and every title has a **?** icon with a
|
|
188
|
+
short help card on what it is for and how to work with it. Every edit lands in the same draft as the source editor and
|
|
189
|
+
rewrites only the bytes it changes (prose, comments and blank lines stay exactly as they are), so `Ctrl/Cmd+S` saves it
|
|
190
|
+
and **Discard** throws the panel edits away too. The model behind it is `lib/case-doc.ts`, the same parser
|
|
191
|
+
`pnpm case:check` uses.
|
|
192
|
+
|
|
193
|
+
The "Jobs" bar at the bottom (opening it shrinks the page above instead of covering it) shows every command started from the UI with live stdout/stderr and its exit
|
|
194
|
+
code; a job can be cancelled there. Only these fixed actions can run (`signin <profile>`, `visit <profile>` and `run` with
|
|
195
|
+
checked options) the three AI commands for an existing case and `/make-case` for a memo (`claude -p` / `opencode run`); there is no free shell. Set `ASRUN_UI_PORT` to change the port.
|
|
196
|
+
|
|
197
|
+
An AI job's log is read, not skimmed: a timestamp in its own style, one line per tool call (its name, what it touched and
|
|
198
|
+
how long it ran), the first lines of what that call returned with the colors a command printed, the agent's thinking
|
|
199
|
+
dimmed and italic while **Thinking** is ticked, and what the agent says rendered as Markdown. The log file kept under
|
|
200
|
+
`.tmp/ui-jobs/` holds the same text with the timestamps written out. Long output is a preview (six lines, then a count),
|
|
201
|
+
so a page snapshot or a big file cannot flood the log. When an AI job finishes, a thin **What to do next** bar appears
|
|
202
|
+
under the log; click it (or **Details**) to open its full text in a sheet from the right. The log keeps its full height,
|
|
203
|
+
so a short Jobs drawer has one scrollbar in the log, not a second one in the hint.
|
|
204
|
+
|
|
205
|
+
The **Model** picker next to the Claude / OpenCode switch chooses the model the AI commands run with. It does not
|
|
206
|
+
carry its own list: it reads the models of the CLI that is selected on this machine, so it shows the providers you
|
|
207
|
+
already signed in and the models your own configuration names.
|
|
208
|
+
- OpenCode: `opencode models`, i.e. every provider in `opencode auth list`, then passed as `--model`. Without a
|
|
209
|
+
model, a one-shot `opencode run` uses OpenCode's **built-in free model**, whose daily limit a long day of runs can
|
|
210
|
+
use up; the picker preselects the model your OpenCode TUI last used. Set `"model"` in `opencode.jsonc` to stop
|
|
211
|
+
depending on this browser's choice.
|
|
212
|
+
- Claude Code: there is no command that lists models, so the rows come from what Claude Code itself knows on this
|
|
213
|
+
machine (`model`, `modelPicker`, `ANTHROPIC_CUSTOM_MODEL_OPTION`, `ANTHROPIC_DEFAULT_*_MODEL` in its settings and
|
|
214
|
+
environment, plus the aliases `sonnet` / `opus` / `haiku` / `fable` / `opusplan`), and, for a gateway set with
|
|
215
|
+
`ANTHROPIC_BASE_URL`, from that gateway's `/v1/models` (the request Claude Code makes when its model discovery is
|
|
216
|
+
on, with your `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` / `ANTHROPIC_CUSTOM_HEADERS`). A gateway that serves
|
|
217
|
+
other models, such as Gemini, shows them too. Cloud providers (Vertex, Bedrock, Foundry) are **not** queried:
|
|
218
|
+
Claude Code does not list their models either, so only the configured ones appear there.
|
|
219
|
+
|
|
220
|
+
The choice is remembered per CLI in the browser and is shown in the command line of every AI run.
|
|
221
|
+
|
|
222
|
+
The **Thinking** icon beside the model picker (a small chip; lit while it is on) passes `--thinking` to OpenCode, so the
|
|
223
|
+
model is asked to think and the job log shows it. It is off by default: thinking costs tokens and makes a run slower.
|
|
224
|
+
Claude Code has no such flag, so the icon is disabled there; it shows thinking when its model and settings have it on.
|
|
225
|
+
Whether anything appears also depends on the model: one that does not return its reasoning shows none.
|
|
226
|
+
|
|
227
|
+
### Checking what a test actually did
|
|
228
|
+
Every run records a video and a trace per test. A new browser tab gets its own recording; the **Reports** page joins
|
|
229
|
+
them in order into one video per test (`reports/<run>/videos/`, needs ffmpeg, see Quick start). The full Playwright report
|
|
230
|
+
still lists the recordings separately. Add `--step-text`
|
|
231
|
+
to `pnpm asrun` to overlay the current step's title on the videos; without it the videos are plain.
|
|
232
|
+
In `pnpm report`, open the test: the steps are named after the steps in the case document. The trace shows each action with
|
|
233
|
+
the page before and after it, plus every network request. `pnpm asrun` also prints the file paths of the
|
|
234
|
+
videos and traces when it finishes; open a trace directly with `pnpm exec playwright show-trace <trace.zip>`.
|
|
235
|
+
Recordings contain real production pages, so do not share them.
|
|
236
|
+
|
|
237
|
+
### Logging in
|
|
238
|
+
`pnpm signin <profile-id>` opens a normal browser with a small card in its bottom-left corner. Log in yourself,
|
|
239
|
+
including 2FA, wait for the logged-in page, then press **Done, save the session** on that card (or press Enter in the
|
|
240
|
+
terminal, which does the same). The session is saved to `.auth/<profile-id>.json` and verified. Nothing about 2FA is
|
|
241
|
+
automated.
|
|
242
|
+
|
|
243
|
+
Login itself is not tested. Before every run, `pnpm asrun` checks the session of each profile of the chosen
|
|
244
|
+
environment. If a profile has no saved session, or the session expired (the check page redirects to `/login`), the
|
|
245
|
+
tests of that profile fail right away (no retries) and the run prints:
|
|
246
|
+
```
|
|
247
|
+
Not logged in (tests for these profiles will FAIL):
|
|
248
|
+
example-production expired -> pnpm signin example-production
|
|
249
|
+
Or skip them for this run: pnpm asrun --env production --skip-profile example-production
|
|
250
|
+
```
|
|
251
|
+
Either log in (`pnpm signin <profile-id>`) or leave the profile out with `--skip-profile`. Profiles that are logged
|
|
252
|
+
in are tested as usual. If a session expires in the middle of a run, the failed test is marked
|
|
253
|
+
"session expired" in the output.
|
|
254
|
+
|
|
255
|
+
Sessions are as sensitive as passwords: `.auth/` (one file per profile) is git-ignored, never share or commit it.
|
|
256
|
+
|
|
257
|
+
## From a case document to a test (AI workflow)
|
|
258
|
+
|
|
259
|
+
0. **(Optional) start from raw notes.** If you only have a memo or a list of things to check, run
|
|
260
|
+
`/make-case path/to/notes.md` (or paste the text after the command). The agent normalizes the text into steps in
|
|
261
|
+
the project's artifact language (English by default), marks steps that write data (`[writes]`) or send email
|
|
262
|
+
(`[email]`), keeps your original lines in a `## Source` table, lists the gaps with a suggested default for each,
|
|
263
|
+
and only then writes `cases/<id>.md`. It never writes specs and
|
|
264
|
+
never invents expected values (unknown ones become a default sentence with a `TODO` block under it). It never opens a
|
|
265
|
+
browser: what the pages really show is found in step 2. `pnpm case:check` validates the front matter of every case.
|
|
266
|
+
1. **Write the case.** `pnpm case:new header-logo` creates `cases/header-logo.md` from a template; `pnpm case:new documents/header-logo` puts it in the
|
|
267
|
+
group folder `cases/documents/` (created on demand, one level only). The id of a case is its path without `.md`, its spec is
|
|
268
|
+
`tests/documents/header-logo.spec.ts`, and its expected values are in its own `## Values` block.
|
|
269
|
+
Fill in steps, what is expected for every profile, and the exact values the spec asserts under Values, one entry per
|
|
270
|
+
profile id (`default` applies to all). Plain language is fine for the checks. Write it in the project's artifact
|
|
271
|
+
language (`AGENTS.md` §2; English by default); if a case is in another language, `/gen-spec` translates the case
|
|
272
|
+
file to the artifact language first (and tells you).
|
|
273
|
+
2. **(Optional) explore the case.** In OpenCode or Claude Code run `/explore-case header-logo production` (the case id is
|
|
274
|
+
required; the environment is optional: `production` or `staging`). This is the only step where a person and the
|
|
275
|
+
agent work together. The agent opens the real pages of the profiles that are logged in and only **reads** them: it
|
|
276
|
+
never creates, edits, publishes, shares or deletes anything, in staging or in production. It fixes wording in the
|
|
277
|
+
case file, writes what the pages show into it (marked `observed, 🔍 to be confirmed by QA`), and turns what it cannot
|
|
278
|
+
know into `🚫 TODO [blocking]` / `❓ TODO [non-blocking]` questions (every marker starts with its emoji). Every question is a block
|
|
279
|
+
under a sentence that is the agent's suggestion, written so that it can stand in the case as it is:
|
|
280
|
+
```
|
|
281
|
+
- Clicking "Edit Draft" opens the document editor, in the same tab or in a new tab.
|
|
282
|
+
- ❓ TODO [non-blocking]: Same tab or new tab? (👁️ not observed)
|
|
283
|
+
- ✍️ Answer:
|
|
284
|
+
```
|
|
285
|
+
It edits only the case file. **To accept the suggestion, delete the whole block** (the `TODO` line and the
|
|
286
|
+
`Answer:` line). **To give your own answer, write it after `Answer:`** (you may delete the rest of the block).
|
|
287
|
+
Then run the command again. It ends with `RESULT: ready` or `RESULT: needs-input`. Under the output of an Explore run,
|
|
288
|
+
a one-line **What to do next** bar shows what is left: the open questions (a red badge when one must be answered),
|
|
289
|
+
what the exploration marked for someone to confirm, and **Generate spec**, which opens the case with that command
|
|
290
|
+
selected. **Details** (or **Answer** when the agent waits) opens a sheet from the right with the question list, the
|
|
291
|
+
answer box and **Use the suggested defaults**, the confirmations and the generate action. A run that ends with
|
|
292
|
+
`needs-input` shows an amber "Needs your input" badge; sending answers (or "Use the suggested defaults", which
|
|
293
|
+
accepts every suggestion) continues the same agent conversation with the same permissions, folds them into the case
|
|
294
|
+
file and ends with a new `RESULT:`. Other commands with open questions show the answer box in the same sheet. A run
|
|
295
|
+
can be continued once per turn, and only while the UI server has not been restarted; after a restart, answer in the
|
|
296
|
+
case file and run the command again.
|
|
297
|
+
|
|
298
|
+
**Attended exploration (`--attended`, or the "Attended" box on the Cases page, ticked by default).** Use it for a case that creates or
|
|
299
|
+
edits data, or needs a choice (an existing document, items). Stay at the computer: the browser window opens and the
|
|
300
|
+
agent waits while you do those steps. The window shows a small card in the bottom-left corner that says what to do (drag it by its title bar or minimize it if it is in the way), with **Done** (hand
|
|
301
|
+
control back) and **Cannot do this**; the agent also continues by itself when the finish condition it named shows up
|
|
302
|
+
on the page (for example a "Saved" message), and the card turns green when control is back. It waits up to 10
|
|
303
|
+
minutes per step. Delete the `[asrun]` items you created before you press Done on the last step. What you did is
|
|
304
|
+
saved in the case under the step as **general recipe lines** (`- Recipe (phase): ...`): positions instead of names
|
|
305
|
+
("select items #1 to #3"), `[asrun]` names for created data, control labels but no selectors. Without `--attended`
|
|
306
|
+
the agent never waits: a step that needs a person becomes a `TODO [blocking]`.
|
|
307
|
+
3. **Log in the profiles the case needs** (`pnpm status`, `pnpm signin ...`). Skip for `login: none` cases.
|
|
308
|
+
4. **Generate.** In OpenCode or Claude Code run `/gen-spec header-logo staging` (the environment is required; there is
|
|
309
|
+
no default). The agent replays the case on the real pages with your saved sessions (it never writes data), writes `tests/header-logo.spec.ts`, fills the
|
|
310
|
+
case's `## Values` for the profiles it explored and checks the files (`pnpm typecheck`, `pnpm case:check`). It does not run the
|
|
311
|
+
spec and does not create screenshot baselines: running specs is a separate step (6). It does not stop to ask: what the
|
|
312
|
+
page answers is written into the case as observed (QA confirms it), and what is still unknown stays in the case as a
|
|
313
|
+
`TODO`. It ends with `RESULT: ready` or `RESULT: needs-input`. A case with a `[writes]` or `[input]` step that has no
|
|
314
|
+
`- Recipe:` line under it is not generated: it stops with `needs-input` and asks you to run step 2 with
|
|
315
|
+
`--attended`.
|
|
316
|
+
If the spec already exists, the command leaves it alone and says so. Add `--force` (`/gen-spec header-logo staging
|
|
317
|
+
--force`) to regenerate it from the case and overwrite it (the case's `## Values` and baseline images are kept; the
|
|
318
|
+
report lists the ones that may be out of date). In the web UI, **Run** on a case that has a spec
|
|
319
|
+
asks first: "Overwrite the spec?". The server refuses to start `gen-spec` on an existing spec without that choice.
|
|
320
|
+
When the run ends, **What to do next** offers **Open Run tests** with the case already selected, for step 6.
|
|
321
|
+
5. **Review what the AI produced.** Check: the spec reads like your case, and the `## Values` block of the case has the
|
|
322
|
+
right values for each profile (you normally don't edit these by hand, but you can).
|
|
323
|
+
6. **Run it:** `pnpm asrun --env <env> --grep <case-id>`. If the spec has visual checks, add `--update-snapshots`
|
|
324
|
+
the first time, then open the new images in `tests/<case>.spec.ts-snapshots/` and confirm each is the correct logo
|
|
325
|
+
for that profile. Baselines are the source of truth for pixel checks. Then run all profiles and `pnpm report`.
|
|
326
|
+
|
|
327
|
+
When the product changes and a test fails, run `/heal-spec <case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]`
|
|
328
|
+
(or **Heal spec** in the Cases toolbar, or **Heal this failure** on a failed result on the Reports page). The agent
|
|
329
|
+
decides whether the test needs updating or whether it found a real problem, and never edits an expectation just to
|
|
330
|
+
make a failure disappear. How it works:
|
|
331
|
+
- The environment is always required. Only the profiles that are logged in there are run (`pnpm asrun --logged-in`
|
|
332
|
+
leaves the others out instead of failing them).
|
|
333
|
+
- The web UI first opens a dialog: the profiles to heal (every logged-in profile ticked) and an optional text box for what
|
|
334
|
+
you want healed or any evidence and clues. The text is stored in `.tmp/hints/` (git-ignored) and passed as `--hints`.
|
|
335
|
+
Clues guide the diagnosis; they never lower the bar for a product finding.
|
|
336
|
+
- The agent reads the existing report first (`pnpm heal:evidence --case <id> --env <env>`: status, failed step, error,
|
|
337
|
+
call log and screenshots per profile, and whether the report is fresh or stale) and runs the case again only when
|
|
338
|
+
there is no report, it is stale, it does not explain the failure, or a clue has to be checked. After it edits the
|
|
339
|
+
spec it runs it once to confirm.
|
|
340
|
+
|
|
341
|
+
### Targets, readiness and downloads (why generated specs can be trusted)
|
|
342
|
+
Three rules keep an AI-written spec from passing or failing for the wrong reason. They hold in exploring, generating and
|
|
343
|
+
healing (`AGENTS.md`, `<framework>/.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`).
|
|
344
|
+
- **Targets.** A check on a logo, image, icon, file, PDF, screenshot or repeated text points to a card under `## Targets`
|
|
345
|
+
in the case (`[T1]`): what it is (and what it is not), where it is, how it is identified (never `.first()` or a bare
|
|
346
|
+
`img`), when it is ready, and a highlighted reference screenshot under `.explore/targets/`. When the agent is in doubt
|
|
347
|
+
it does not pick: it writes a `🚫 TODO [blocking]` that asks for the exact image, text, place or UI, with the
|
|
348
|
+
candidates listed. Answer it by editing the case (or, in an attended run, by outlining the target in the browser).
|
|
349
|
+
- **Readiness.** After an action that loads something, the key requests must finish and the content must be visible before
|
|
350
|
+
anything is read. Specs use `actAndWait` (`@asrunjs/asrun/ready`); "missing" and "empty" checks need a positive ready signal.
|
|
351
|
+
Healing treats a "missing / empty / timeout" failure as a readiness problem first.
|
|
352
|
+
- **Downloads.** `verifyDownload` (`@asrunjs/asrun/downloads`) checks that the download starts, finishes, the file exists and the
|
|
353
|
+
file is valid for its kind; any failure or timeout fails the whole step with a message that names the stage.
|
|
354
|
+
- **PDFs are read like a person reads them.** `expectPdf` (`@asrunjs/asrun/pdf`) renders the pages to images (a logo region must not
|
|
355
|
+
be blank and must match the profile's baseline, 2% tolerance) and extracts the text (for example, no other profile's name).
|
|
356
|
+
Every page is checked unless the case limits the pages in strong words ("only", "exactly", "must"); "should",
|
|
357
|
+
"always", "usually" or no wording mean every page.
|
|
358
|
+
`pnpm pdf:render <file.pdf> --out .explore/targets/<case-id>/<name> --corner top-left` lets you (or the exploring agent)
|
|
359
|
+
look at the rendered pages.
|
|
360
|
+
- **Checks you can run yourself:** `pnpm spec:lint` (fails on screenshots without a target, `.first()` on images, sleeps,
|
|
361
|
+
`networkidle`, raw download events, a `%PDF` check on its own), `pnpm baselines:review` (opens nothing; lists every
|
|
362
|
+
baseline with the card that says what it must show, so you can look at each new image), and `pnpm case:check`
|
|
363
|
+
(fails on an incomplete card, warns on an image check with no card).
|
|
364
|
+
|
|
365
|
+
### Updating baselines on purpose
|
|
366
|
+
If a profile really changed its logo, or a baseline is wrong: fix the spec first (the check must look at the element
|
|
367
|
+
its target card names; a wrong locator just makes a new wrong image), then
|
|
368
|
+
`pnpm asrun --env <env> --profile <id> --grep <case> --update-baselines <case-id>[,<case-id>]`, then look at the new
|
|
369
|
+
image before keeping it (`pnpm baselines:review <case-id>`).
|
|
370
|
+
- `--update-baselines` names the cases whose images are overwritten (`--update-snapshots=all`). They must be among the
|
|
371
|
+
cases the filters run. Playwright has one global switch, so they run as a second pass with their own report
|
|
372
|
+
(`<run-id>-baselines`); every other case is compared with its baselines as usual.
|
|
373
|
+
- Without `--update-baselines`, `--update-snapshots` still works as before, but it rewrites every image of the run.
|
|
374
|
+
- In the web UI, every case in **Cases to run** has an **Update baselines** box (nothing is ticked by default). A
|
|
375
|
+
confirm dialog follows, and when the run ends the job output shows each rewritten image next to the target card that
|
|
376
|
+
says what it must show.
|
|
377
|
+
- Not part of exploring, generating or healing: an agent never updates a baseline.
|
|
378
|
+
|
|
379
|
+
### Suites
|
|
380
|
+
Each case lists its suites in the front matter: `suites: [smoke, regression]` (any lowercase words). A case runs in
|
|
381
|
+
every suite it lists. The generated spec tags itself `@<suite>` for each one, which:
|
|
382
|
+
- shows as badges next to the test in the terminal and in `pnpm report`, and can be typed into the report's
|
|
383
|
+
search box (`@smoke`) to filter there;
|
|
384
|
+
- drives `pnpm asrun --suite <name>`, which can be combined with `--profile` and `--grep`. A case that is in two
|
|
385
|
+
of the chosen suites still runs once.
|
|
386
|
+
|
|
387
|
+
An unknown suite is rejected with the list of suites the cases use. The old `priority: <level>` is read as
|
|
388
|
+
`suites: [<level>]`.
|
|
389
|
+
|
|
390
|
+
### Other front matter
|
|
391
|
+
- `writes: [staging]` or `writes: [staging, production]` (omit it for a read-only case): the environments in which the
|
|
392
|
+
case may create or change data of the test accounts (and cleans it up). It then needs
|
|
393
|
+
`writes-note: <what it creates or changes>`. In an environment that is not listed, the case is **skipped with a
|
|
394
|
+
notice**, not failed. The old `writes: test-data` is refused with this hint.
|
|
395
|
+
- `envs: [production, staging]` (optional): the case runs only in these environments; in others it is skipped. Every
|
|
396
|
+
environment in `writes` must also be in `envs`.
|
|
397
|
+
|
|
398
|
+
Cases that write data get a red **WRITES** badge in the web UI and the report, and an `@writes` tag.
|
|
399
|
+
|
|
400
|
+
## What a case may and may not do
|
|
401
|
+
Tests run against production and staging. By default a case is read-only: look, read, compare. No creating,
|
|
402
|
+
editing, deleting, submitting or accepting anything.
|
|
403
|
+
|
|
404
|
+
A case may write only in the environments listed in its `writes` front matter (elsewhere it is skipped with a notice),
|
|
405
|
+
and only data of the test account (named with the prefix `[asrun]`), which it must clean up.
|
|
406
|
+
**AI agents never write data while they explore or generate**, in either environment: a person does those steps, in
|
|
407
|
+
the browser, in an attended exploration (step 2 above).
|
|
408
|
+
- **Staging:** no confirmation needed.
|
|
409
|
+
- **Production (and any environment that is not staging)**, for a case that lists it under `writes`: a person confirms every run. `pnpm asrun --env production` lists
|
|
410
|
+
the write cases and asks you to type `production`. The web UI shows a confirm dialog. Unattended runs use
|
|
411
|
+
`--allow-writes=production`; AI agents never do. `--read-only` leaves the write cases out. The permission is also
|
|
412
|
+
enforced inside the test itself, so running `playwright test` directly cannot bypass it.
|
|
413
|
+
- **Cleanup:** data is named `[asrun] ...`, registered with the `records` fixture and removed after the test, also
|
|
414
|
+
after a failure. Whatever could not be removed is shown in red in the report and listed by
|
|
415
|
+
`pnpm cleanup --env <env>`; delete it by hand, then `pnpm cleanup --env <env> --forget`.
|
|
416
|
+
- **No retries, one at a time:** write cases are never retried, and run one at a time per profile.
|
|
417
|
+
- **Email:** never sent from production. In other environments only if the profile has `email: true` in `profiles.yaml`.
|
|
418
|
+
|
|
419
|
+
## Author
|
|
420
|
+
Created by Zoro Chen.
|
|
421
|
+
|
|
422
|
+
## Layout
|
|
423
|
+
|
|
424
|
+
**In a project that uses the framework** (everything here is committed except the ignored folders):
|
|
425
|
+
|
|
426
|
+
```
|
|
427
|
+
profiles.yaml one entry per environment: URL, account label, tags, environment (no expected values)
|
|
428
|
+
cases/ human-written cases (+ _template.md); optional group folders: cases/<group>/<name>.md
|
|
429
|
+
tests/ generated specs (same folders as cases/), and *-snapshots/ baseline images per profile
|
|
430
|
+
tsconfig.json plain compiler options: specs import the framework by package name (`@asrunjs/asrun/fixtures`)
|
|
431
|
+
playwright.config.ts re-exports node_modules/@asrunjs/asrun/dist/playwright.config.js
|
|
432
|
+
AGENTS.md the project rules: they point at the framework baseline, and §2 says what differs (narrows or replaces)
|
|
433
|
+
.claude/commands/ /make-case, /explore-case, /gen-spec, /heal-spec: thin shells that load the framework skill (Claude Code)
|
|
434
|
+
.opencode/commands/ the same commands for OpenCode (it does not read `.claude/commands`)
|
|
435
|
+
.claude/skills/ optional project overrides of a skill: <skill>.md, read by the command and the skill (not a discovered skill)
|
|
436
|
+
.mcp.json, opencode.jsonc the playwright-test MCP server, for Claude Code and OpenCode
|
|
437
|
+
.env ASRUN_ROOT (which workspace the commands use) and local settings (git-ignored)
|
|
438
|
+
node_modules/@asrunjs/asrun/ the framework (installed dependency; never edited)
|
|
439
|
+
.auth/ saved logins, one per profile (git-ignored)
|
|
440
|
+
reports/ one folder per run; reports/latest points to the newest (git-ignored)
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
**In the framework repository** (no workspace content at the root: no `profiles.yaml`, `cases/`, `tests/` or reports):
|
|
444
|
+
|
|
445
|
+
```
|
|
446
|
+
lib/ fixtures, session guard, image assertions, case values (imported as asrun/<module>)
|
|
447
|
+
scripts/ the pnpm commands above, plus init (workspace files) and doctor (checks)
|
|
448
|
+
templates/ what `pnpm asrun init` writes into a project
|
|
449
|
+
ui/ local web UI: server/ (Hono API, job runner), web/ (React + Vite), shared/ (types)
|
|
450
|
+
docs/ static docs; theme-guide.html = the Inkstone theme (designed and created by Zoro Chen): colors, buttons, icons of the web UI (open in a browser, or via the "Inkstone theme" link in the UI's top bar)
|
|
451
|
+
examples/demo/ a committed workspace for developing the framework (public demo site; .auth/ and reports stay local)
|
|
452
|
+
.claude/ skills + slash commands for Claude Code; the skills are shared with OpenCode
|
|
453
|
+
.opencode/commands/ the same commands for OpenCode (it does not read `.claude/commands`)
|
|
454
|
+
AGENTS.md the product baseline rules (OpenCode reads it); CLAUDE.md imports it for Claude Code
|
|
455
|
+
docs/agents/ framework-dev.md: how to change asrun itself, never used by a project
|
|
456
|
+
opencode.jsonc MCP config for OpenCode; .mcp.json is the same server for Claude Code
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
## Developing the framework
|
|
460
|
+
|
|
461
|
+
`pnpm go`, `pnpm asrun`, `pnpm ui` work here too: without a workspace they use `examples/demo`. `init` writes the
|
|
462
|
+
framework's `.env` for that (`ASRUN_ROOT=examples/demo`), so the commands need no environment variable:
|
|
463
|
+
|
|
464
|
+
```bash
|
|
465
|
+
pnpm asrun init --root examples/demo # creates the demo workspace and the framework-root .env
|
|
466
|
+
pnpm asrun --env staging --read-only
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
The demo is a workspace of its own as well:
|
|
470
|
+
|
|
471
|
+
```bash
|
|
472
|
+
cd examples/demo && ASRUN_ROOT=. pnpm asrun doctor
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
`ASRUN_ROOT` still wins over `.env` (a real environment variable always does), so one checkout can be pointed at
|
|
476
|
+
another project for a single command:
|
|
477
|
+
|
|
478
|
+
```bash
|
|
479
|
+
ASRUN_ROOT=../my-real-project pnpm asrun --env staging --read-only
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
Two paths keep the two halves apart: `ROOT` (`lib/paths.ts`) is the workspace, `FRAMEWORK_ROOT` is this folder. The
|
|
483
|
+
package is compiled to `dist/` (`pnpm build`) and the web UI to `ui/dist` (`pnpm ui`); both are shipped by `prepack`.
|
|
484
|
+
Nothing writes into `FRAMEWORK_ROOT` at run time. After changing `lib/`, run `pnpm build` before exercising the demo's
|
|
485
|
+
specs (they import the package by name, which resolves to `dist/`). After changing `templates/`, bump `version` in
|
|
486
|
+
`package.json`; `pnpm asrun doctor` in a project then shows that it is behind.
|
|
487
|
+
|
|
488
|
+
## How profiles become tests
|
|
489
|
+
Every entry in `profiles.yaml` becomes a Playwright project (name = profile id) with its own `baseURL` and saved login.
|
|
490
|
+
A spec is written once and runs once per profile; the `profile` fixture exposes that profile's `id`, `env`, `baseURL`,
|
|
491
|
+
`account`, `tags` and `email`. The expected values come from the case document, through
|
|
492
|
+
`requireValues(caseId, profile)` (its `## Values` block, `default` merged with the profile's own entry). Adding a
|
|
493
|
+
profile = add a YAML entry (by hand or with the form on the Profiles page) + `pnpm signin <id>`, then add the values
|
|
494
|
+
for it to the cases (see the guide on the Profiles page).
|
|
495
|
+
|
|
496
|
+
## Troubleshooting
|
|
497
|
+
- `Profile "x" is not logged in`: `pnpm signin <profile-id>`, or run with `--skip-profile <profile-id>`.
|
|
498
|
+
- Screenshot mismatch: open the report; the diff view shows expected / actual / diff. If the change is
|
|
499
|
+
intended, update the baseline as above.
|
|
500
|
+
- Report empty or missing: run `pnpm asrun --env <env>` first.
|
|
501
|
+
- `Choose the environment to test`: there is no default; add `--env production` or `--env staging`.
|
|
502
|
+
- An AI command says the `playwright-test` MCP server is missing: run `pnpm asrun doctor`. OpenCode reads only
|
|
503
|
+
`opencode.jsonc` and Claude Code only `.mcp.json`; `pnpm asrun init` writes both, and `opencode mcp list` /
|
|
504
|
+
`claude mcp list` shows whether the server started (`failed: ... path specified` means the `playwright` path in
|
|
505
|
+
that file does not resolve: `pnpm asrun init --force <file>`).
|
|
506
|
+
- A project rule does nothing: `AGENTS.md` §2 needs the rule id and an `Effect` of `narrows` or `replaces`; `F1`..`F6`
|
|
507
|
+
cannot be changed. `pnpm asrun doctor` names a wrong id, a missing `Effect` or a non-negotiable rule.
|
|
508
|
+
- A skill override does nothing: it is `.claude/skills/<skill>.md` (the skill name, for example `generating-specs`,
|
|
509
|
+
not the command name) and it is not discovered as a skill: the command and the skill read it. `doctor` lists the
|
|
510
|
+
overrides it sees.
|
|
511
|
+
- `pnpm asrun init --force AGENTS.md` is refused: `AGENTS.md`, `CLAUDE.md` and `.claude/skills/**` belong to the
|
|
512
|
+
project. Add `--reset-rules` to replace them with the template, and expect to lose the project's rules.
|