@asrunjs/asrun 0.8.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. package/.claude/commands/explore-case.md +14 -5
  2. package/.claude/commands/gen-spec.md +12 -5
  3. package/.claude/commands/heal-spec.md +23 -8
  4. package/.claude/commands/make-case.md +1 -1
  5. package/.claude/skills/exploring-cases/SKILL.md +82 -37
  6. package/.claude/skills/exploring-cases/references/assertion-targets.md +8 -4
  7. package/.claude/skills/exploring-cases/references/case-rules.md +110 -0
  8. package/.claude/skills/exploring-cases/references/handoff-card.js +73 -11
  9. package/.claude/skills/exploring-cases/references/open-questions.md +60 -8
  10. package/.claude/skills/exploring-cases/references/readiness.md +4 -3
  11. package/.claude/skills/exploring-cases/references/tools.md +117 -0
  12. package/.claude/skills/exploring-cases/references/user-handoff.md +46 -17
  13. package/.claude/skills/generating-specs/SKILL.md +148 -55
  14. package/.claude/skills/generating-specs/references/conventions.md +4 -0
  15. package/.claude/skills/making-cases/SKILL.md +14 -7
  16. package/.claude/skills/making-cases/references/gaps.md +7 -4
  17. package/.claude/skills/making-cases/references/normalizing.md +1 -1
  18. package/AGENTS.md +68 -20
  19. package/README.md +8 -6
  20. package/bin/asrun.mjs +1 -0
  21. package/dist/lib/case-doc.d.ts +41 -4
  22. package/dist/lib/case-doc.js +109 -17
  23. package/dist/scripts/check-cases.js +89 -1
  24. package/dist/scripts/doctor.js +50 -3
  25. package/dist/scripts/markers-confirm.d.ts +1 -0
  26. package/dist/scripts/markers-confirm.js +139 -0
  27. package/dist/ui/server/actions.js +57 -18
  28. package/dist/ui/server/app.js +7 -2
  29. package/dist/ui/server/job-store.d.ts +60 -0
  30. package/dist/ui/server/job-store.js +198 -0
  31. package/dist/ui/server/jobs.d.ts +46 -4
  32. package/dist/ui/server/jobs.js +361 -59
  33. package/dist/ui/server/process-info.d.ts +17 -0
  34. package/dist/ui/server/process-info.js +126 -0
  35. package/dist/ui/server/result-state.d.ts +10 -0
  36. package/dist/ui/server/result-state.js +106 -0
  37. package/dist/ui/shared/ai-commands.d.ts +9 -3
  38. package/dist/ui/shared/ai-commands.js +25 -11
  39. package/dist/ui/shared/types.d.ts +18 -3
  40. package/docs/agents/case-format.md +39 -5
  41. package/docs/ai-workflow.md +80 -21
  42. package/docs/cases.md +28 -4
  43. package/docs/commands.md +6 -1
  44. package/docs/theme-guide.html +93 -5
  45. package/docs/troubleshooting.md +47 -3
  46. package/docs/web-ui.md +55 -10
  47. package/package.json +2 -1
  48. package/templates/.claude/commands/explore-case.md +14 -4
  49. package/templates/.claude/commands/gen-spec.md +12 -4
  50. package/templates/.claude/commands/heal-spec.md +27 -13
  51. package/templates/.claude/commands/make-case.md +1 -1
  52. package/templates/AGENTS.md +9 -6
  53. package/templates/cases/_template.md +7 -0
  54. package/templates/package.json +1 -0
  55. package/ui/dist/assets/{dist-ChBHX4rh.js → dist-2msWI-5M.js} +1 -1
  56. package/ui/dist/assets/{dist-DoChb6QQ.js → dist-4U_gPHXf.js} +1 -1
  57. package/ui/dist/assets/{dist-CJiwxr1r.js → dist-BMfJatHo.js} +1 -1
  58. package/ui/dist/assets/{dist-N-o4QIaZ.js → dist-BVvHVJ3N.js} +1 -1
  59. package/ui/dist/assets/{dist-BC_P1Gnf.js → dist-BmWaNjqO.js} +1 -1
  60. package/ui/dist/assets/{dist-Do_MhHKF.js → dist-BwNz_xW2.js} +1 -1
  61. package/ui/dist/assets/{dist-D6jmnDr-.js → dist-J36xayOv.js} +1 -1
  62. package/ui/dist/assets/{dist-CavmZIv7.js → dist-e-KhTWHJ.js} +1 -1
  63. package/ui/dist/assets/{dist-BpkTvixI.js → dist-sfZKBSMm.js} +1 -1
  64. package/ui/dist/assets/{dist-BgPbx2GZ.js → dist-wjB7g6OJ.js} +1 -1
  65. package/ui/dist/assets/index-CGbRJT3b.js +348 -0
  66. package/ui/dist/assets/index-CI0i0LWc.css +2 -0
  67. package/ui/dist/help/case-explore.png +0 -0
  68. package/ui/dist/index.html +2 -2
  69. package/ui/dist/assets/index-Cy2l91qV.css +0 -2
  70. package/ui/dist/assets/index-DGQ5logJ.js +0 -338
@@ -8,16 +8,31 @@ so the procedure lives in one place and a project can refine it (see
8
8
  | Command | What it does |
9
9
  |---|---|
10
10
  | `/make-case <notes-or-path>` | Turn raw notes into case documents. Optional step. |
11
- | `/explore-case <case-id> <env> [--attended]` | Open the real pages and write what they show into the case. |
12
- | `/gen-spec <case-id> <env> [--force]` | Write `tests/<case-id>.spec.ts` and fill the case's `## Values`. |
13
- | `/heal-spec <case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]` | Diagnose a failing spec from its last report and fix it. |
11
+ | `/explore-case <case-id> <env> [--attended] [--profiles a,b] [--hints <file>]` | Open the real pages and write what they show into the case. |
12
+ | `/gen-spec <case-id> <env> [--force] [--hints <file>]` | Write `tests/<case-id>.spec.ts` and fill the case's `## Values`. |
13
+ | `/heal-spec <case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]` | Repair a failing spec, or refine a case and its spec on request, from the last report and what you asked for. |
14
14
 
15
15
  The environment is **required** by every command except `/make-case`, which does not open pages. There is no default.
16
16
 
17
- Every command ends with `RESULT: ready` or `RESULT: needs-input`. `needs-input` means somebody has to answer a
17
+ `--profiles` exists on `/explore-case` and `/heal-spec`, not on `/gen-spec`: a spec runs once per profile in the
18
+ environment, so it is generated from every profile the case holds information for, never from a chosen subset.
19
+
20
+ `--hints <file>` (or free text after the arguments) is optional on all three commands that read a case: what you know,
21
+ want checked or want changed. It guides that run. After the work, a reusable instruction that is **method**, not an
22
+ expectation, is written into the case's `## Rules`; a one-off clue stays in the report. See
23
+ [Rules for one case](#rules-for-one-case).
24
+
25
+ Every command ends with `RESULT: ready` or `RESULT: needs-input` as plain text on its last line (never in a code fence
26
+ or in emphasis: the web UI reads that line to show the outcome). `needs-input` means somebody has to answer a
18
27
  question in the case file (or, in the web UI, in the job's **Answer** sheet). A one-shot run cannot ask in the chat:
19
28
  it writes its questions into the case and stops.
20
29
 
30
+ A one-shot run is a **build** run: it writes the case or the spec, never a plan. AsRun starts OpenCode with
31
+ `--agent build`, because OpenCode keeps the agent (build or plan) on the session and a plan session would only propose
32
+ changes and ask a question nobody can answer. Whatever mode a run finds itself in, it still ends with the `RESULT:`
33
+ line: a run that was not allowed to write files (plan mode, a read-only sandbox) is `needs-input`, not a plan and a
34
+ question.
35
+
21
36
  In the web UI you never type a command: choose the step in a case toolbar and press **Run AI**, or paste notes on
22
37
  **Cases > From memo**. The UI runs the selected AI tool and streams the output into the Jobs bar.
23
38
 
@@ -40,6 +55,19 @@ It writes `cases/<id>.md` only, and never overwrites an existing case. `pnpm cas
40
55
  `pnpm case:new header-logo` creates `cases/header-logo.md` from the template, or use a case you already have. The
41
56
  format is in [cases.md](./cases.md). Write the steps, the expectations and — when you know them — the values.
42
57
 
58
+ ### Rules for one case
59
+
60
+ A case can hold a `## Rules` section: case-local instructions on **how** to run it (dismiss a cookie banner first, wait
61
+ for a dialog after a request, never use an entry that needs 2FA). It is the place to talk to the agent without putting
62
+ a requirement into the wrong section. One bullet per rule; `Step 3:`, `Before step 3:` or `After step 3:` ties it to a
63
+ step, and `pnpm case:check` fails when that step does not exist.
64
+
65
+ Rules never say what the case proves, never replace a step or an expected value, and never allow writing data, sending
66
+ email or doing a person's step: those stay in `## Expected`, `## Values` and tagged steps with a recipe. The AI reads
67
+ and applies every rule in every run, re-checks them against the page, and lists them in its report. `/explore-case`,
68
+ `/gen-spec` and `/heal-spec` re-check the rules and never delete or rewrite one on their own; a rule that no longer
69
+ holds becomes a question for you.
70
+
43
71
  ## 2. Explore the case (only where a person and the agent work together)
44
72
 
45
73
  ```
@@ -49,12 +77,19 @@ format is in [cases.md](./cases.md). Write the steps, the expectations and — w
49
77
  The agent opens the real pages of the profiles that are logged in and only **reads** them. It never creates, edits,
50
78
  publishes, shares or deletes anything, in staging or in production. It:
51
79
 
52
- - fixes wording in the case file and writes what the pages show into it, marked `observed, 🔍 to be confirmed by QA`;
80
+ - fixes wording in the case file and writes what the pages show into it: a plain note `(seen on <profile>)` when the
81
+ run checks it, and `(observed on <profile>, 🔍 to be confirmed: <what a person must decide>)` only for a line that
82
+ would pass even if it were wrong;
53
83
  - turns what it cannot know into `🚫 TODO [blocking]` / `❓ TODO [non-blocking]` questions, each under a sentence that
54
84
  is its suggestion and can stand in the case as it is, with a last line `- ✍️ Answer:`;
55
85
  - lists the candidates instead of guessing a target (`## Targets`);
56
86
  - edits the case file only (plus reference screenshots under `.explore/targets/`).
57
87
 
88
+ With many profiles, `--profiles a,b` opens only those (ids from `profiles.yaml`, comma-separated). The others are not
89
+ opened and nothing is written about them, so a case can be explored in passes: A and B first, C later. Runs add up; the
90
+ case keeps what the earlier runs recorded. In the web UI the profile picker of the toolbar does the same: every profile
91
+ that is logged in in that environment is ticked, and the list you tick is remembered for the next run there.
92
+
58
93
  **To accept the suggestion, delete the whole block.** **To answer, write after `Answer:`.** Then run the command
59
94
  again. In the web UI, the run's **What to do next** bar offers **Answer**: the question list, an answer box,
60
95
  **Use the suggested defaults** and **Generate spec**. A continued job keeps the same conversation and permissions, and
@@ -68,7 +103,9 @@ the browser window opens and the agent waits while you do those steps.
68
103
  - The window shows a small draggable card in the bottom-left corner with what to do, **Done** (hand control back) and
69
104
  **Cannot do this**. The agent also continues by itself when the finish condition it named shows up (for example a
70
105
  "Saved" message); the card turns green when control is back. It waits up to 10 minutes per step.
71
- - What you did is recorded in the case under the step as **general recipe lines**
106
+ - Recording starts and stops by itself. If Playwright's own recording toolbar shows at the top of the page, leave it
107
+ alone: it can stop the recording.
108
+ - The card records what you did, and the agent writes it into the case under the step as **general recipe lines**
72
109
  (`- Recipe (phase): ...`): positions instead of names of changeable data, `[asrun]` names for created data, control
73
110
  labels but no selectors.
74
111
  - Delete the `[asrun]` items you created before you press Done on the last step.
@@ -92,8 +129,14 @@ Skip this for a `login: none` case. Sessions are per profile; see [profiles.md](
92
129
  ```
93
130
 
94
131
  The agent replays the case on the real pages with your saved sessions (it never writes data) and writes
95
- `tests/header-logo.spec.ts`. It fills the case's `## Values` for the profiles it explored and checks the files
96
- (`pnpm typecheck`, `pnpm case:check`; `pnpm spec:lint <case-id>` must have no failure).
132
+ `tests/header-logo.spec.ts`. It fills the case's `## Values` for the profiles the case holds information for and checks
133
+ the files (`pnpm typecheck`, `pnpm case:check`; `pnpm spec:lint <case-id>` must have no failure).
134
+
135
+ Its profile scope comes from the case, not from a selection: every profile id the case names — the keys of
136
+ `## Values`, a target card's `Reference <profile-id>:` line, an observation — plus `default`. That is why one run per
137
+ subset adds up: explore A and B first and C later, and `/gen-spec` covers all three. A profile the case says nothing
138
+ about runs on `default` alone, and the report names it. A profile that is not logged in is not dropped: its part is
139
+ written from the case and marked `not verified on the page`.
97
140
 
98
141
  It does **not** run the spec and does **not** create screenshot baselines: running is a separate step (6).
99
142
 
@@ -125,7 +168,7 @@ profile. Baselines are the source of truth for pixel checks.
125
168
  Then run all profiles and read the report: the video of each test, screenshots on failure, and the trace with the page
126
169
  and the network requests. See [commands.md](./commands.md) for filters and baselines.
127
170
 
128
- ## Healing a failing spec
171
+ ## Healing or refining a spec
129
172
 
130
173
  ```
131
174
  /heal-spec header-logo staging
@@ -133,14 +176,17 @@ and the network requests. See [commands.md](./commands.md) for filters and basel
133
176
 
134
177
  or **Heal spec** in the case toolbar, or **Heal this failure** on a failed result of the Reports page.
135
178
 
136
- The agent decides whether the test needs updating or whether it found a real product problem. It **never** edits an
137
- expectation just to make a failure disappear.
179
+ The command does two jobs: it repairs a spec that fails, and it refines a case and its spec when you ask for an
180
+ improvement. A passing last run is not a reason to refuse a request. It decides whether the test needs updating or
181
+ whether it found a real product problem, and it **never** edits an expectation just to make a failure disappear.
138
182
 
139
183
  - The environment is required. Only the profiles that are logged in there run (`--logged-in` leaves the others out
140
184
  instead of failing them).
141
- - The web UI first opens a dialog: which profiles to heal (every logged-in profile ticked) and an optional text box
142
- for what you want healed, with evidence or clues. The text is stored in `.tmp/hints/` and passed as `--hints`.
143
- Clues guide the diagnosis; they never lower the bar for a product finding.
185
+ - The web UI first opens a dialog: which profiles to run (every logged-in profile ticked) and an optional text box
186
+ for what should be repaired or improved, with evidence. The text is stored in `.tmp/hints/` and passed as `--hints`.
187
+ It guides the work; it never lowers the bar for a product finding. A reusable instruction that is method rather than
188
+ an expectation is also written into the case's `## Rules` (see [Rules for one case](#rules-for-one-case)); a one-off
189
+ clue stays in the report.
144
190
  - It reads the existing report first:
145
191
 
146
192
  ```bash
@@ -149,13 +195,24 @@ expectation just to make a failure disappear.
149
195
 
150
196
  It prints status, failed step, error, call log and screenshots per profile, and whether the report is fresh or
151
197
  stale. The case is run again only when there is no report, the report is stale, it does not explain the failure, or
152
- a clue has to be checked. After editing the spec, it runs the case once to confirm.
198
+ a request has to be checked. It reads the pages (read-only) when a change has to be judged on them, and it never
199
+ performs a step that writes data. After editing the spec, it runs the case once to confirm.
200
+ - The case is the source of truth: a change to a step, an expectation, a value or a target is written into the case
201
+ first, then into the spec. A large change regenerates the spec from the case.
202
+ - **Review gate.** If the change leaves an open question or a `🔍 to be confirmed` marker in the case, healing stops
203
+ before the spec: you review the case, answer what it asks or settle the marker (the panel shows why it needs you, and
204
+ `pnpm markers:confirm` settles the ones a passing run already checks), then run `/explore-case` again. The spec is
205
+ generated after that, not before. A plain `(seen on <profile>)` note never stops it: the run checks those. A marker
206
+ inside `## Rules` does not stop the spec either, unless it covers a literal the spec asserts.
153
207
 
154
208
  ## Why a generated spec can be trusted
155
209
 
156
- Three binding rules hold in exploring, generating and healing. They live in the framework's `AGENTS.md` and in
157
- `.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`.
210
+ Several binding rules hold in exploring, generating and healing. They live in the framework's `AGENTS.md` and in
211
+ `.claude/skills/exploring-cases/references/case-rules.md`, `assertion-targets.md` and `readiness.md`.
158
212
 
213
+ - **Case rules.** A `## Rules` bullet says how to run the case, never what it proves. The agent applies it while
214
+ exploring and generating, marks code that exists only because of a rule with `// rule: <short quote>`, and lists every
215
+ rule in its report. It never becomes an assertion and never authorizes a write or a person's step.
159
216
  - **Targets.** A check on a logo, image, icon, file, PDF, screenshot or repeated text points to a `[T1]` card under
160
217
  `## Targets`: what it is (and what it is not), where it is, how it is identified (never `.first()` or a bare `img`),
161
218
  when it is ready, and a highlighted reference screenshot under `.explore/targets/`. When in doubt the agent does not
@@ -177,7 +234,8 @@ Checks you can run yourself:
177
234
  |---|---|
178
235
  | `pnpm spec:lint [case-id]` | fails on screenshots without a target, `.first()` on images, sleeps, `networkidle`, raw download events, a `%PDF` check on its own |
179
236
  | `pnpm baselines:review [case-id] [--new]` | lists every baseline with the card that says what it must show, so you can look at each new image |
180
- | `pnpm case:check [case-id]` | fails on an incomplete card or bad values; warns on an image check with no card |
237
+ | `pnpm case:check [case-id]` | fails on an incomplete card, bad values, a marker inside a question or a question under a marked line; warns on an image check with no card and on a marker that does not say what to decide |
238
+ | `pnpm markers:confirm --case <id> --env <env> [--report <run-id>]` | after a passing run, removes the `🔍 to be confirmed` markers whose line the spec checks with `// unconfirmed:`; `--dry-run` lists them |
181
239
  | `pnpm pdf:render <file.pdf> --out <dir> [--pages all] [--corner top-left]` | renders the pages to images and prints the text, to look at a PDF by hand |
182
240
 
183
241
  ## Customizing the prompts
@@ -198,11 +256,12 @@ A project refines a skill with a **flat override file**, named after the skill (
198
256
  The command and the skill read it; it is not discovered as a standalone skill. Rules combine in this order, and a
199
257
  later one wins where they conflict:
200
258
 
201
- 1. `F1`–`F6` of the framework baseline: safety, in every project. Nothing can change them.
259
+ 1. `F1`–`F6` and `R2` of the framework baseline: safety, in every project. Nothing can change them.
202
260
  2. The framework baseline, `<framework>/AGENTS.md`.
203
261
  3. The framework skill.
204
262
  4. The project override `.claude/skills/<skill>.md`.
205
263
  5. The project's own `AGENTS.md` §2 (Project rules).
264
+ 6. A case's `## Rules`: how to run that one case, never what is true.
206
265
 
207
266
  An override is plain Markdown; write only what differs:
208
267
 
@@ -214,8 +273,8 @@ An override is plain Markdown; write only what differs:
214
273
  - The staging account has no published documents: assert the empty state there instead.
215
274
  ```
216
275
 
217
- The project's `AGENTS.md` §2 is the other place: a table of rule ids (`F7`–`F10`; `F1`–`F6` are fixed) with `narrows`
218
- (stricter) or `replaces` (a different default). For example, a project that writes cases in Simplified Chinese
276
+ The project's `AGENTS.md` §2 is the other place: a table of rule ids (`F7`–`F10`, `R1`, `R3`, `R4`; `F1`–`F6` and
277
+ `R2` are fixed) with `narrows` (stricter) or `replaces` (a different default). For example, a project that writes cases in Simplified Chinese
219
278
  `replaces` `F10`. `pnpm asrun doctor` fails on an unknown id, a missing `Effect`, or a non-negotiable rule, and lists
220
279
  the overrides it sees.
221
280
 
package/docs/cases.md CHANGED
@@ -31,6 +31,9 @@ suites: [smoke, regression]
31
31
  ## Goal
32
32
  Why this case exists, in one or two sentences.
33
33
 
34
+ ## Rules
35
+ - Before step 1: dismiss the cookie banner; it covers the header.
36
+
34
37
  ## Steps
35
38
  1. Open `/`.
36
39
  - Ready when: the header has rendered its menu.
@@ -63,8 +66,8 @@ example-staging:
63
66
  Anything a reviewer should know.
64
67
  ````
65
68
 
66
- The headings are fixed English tokens (`Goal`, `Steps`, `Expected`, `Values`, `Visual checks`, `Targets`, `Notes`,
67
- and `Source` when the case was made from raw notes). A project language never translates them.
69
+ The headings are fixed English tokens (`Goal`, `Rules`, `Steps`, `Expected`, `Values`, `Visual checks`, `Targets`,
70
+ `Notes`, and `Source` when the case was made from raw notes). A project language never translates them.
68
71
 
69
72
  ## Front matter
70
73
 
@@ -80,6 +83,21 @@ and `Source` when the case was made from raw notes). A project language never tr
80
83
 
81
84
  `priority: <level>` is the old name of `suites`; it is read as `suites: [<level>]`. Do not use both.
82
85
 
86
+ ## Rules
87
+
88
+ Optional, case-local instructions for the AI agents that explore, generate and heal this case: **how** to run it, never
89
+ **what it proves**. One bullet per rule. A bullet may start with `Step 3:`, `Before step 3:` or `After step 3:` to tie
90
+ it to that step; `pnpm case:check` fails when the step does not exist. A rule without a prefix applies to the whole case.
91
+
92
+ - Use it for a page quirk, a mandatory read-only preparation (dismiss a cookie banner), a control to avoid, or a
93
+ waiting/attention hint.
94
+ - Do not put an expectation, a literal or a target here: those stay in `## Expected`, `## Values` and `## Targets`.
95
+ - A rule never allows creating, changing or deleting data, sending email or doing a person's step. Those stay tagged
96
+ steps with a recipe.
97
+ - A rule that conflicts with the rest of the case is a blocking question, not an override. `/explore-case` re-checks
98
+ each rule against the page and never deletes one silently.
99
+ - The framework baseline's `R1`–`R4` define exactly what a rule may say.
100
+
83
101
  ## Steps
84
102
 
85
103
  Numbered lines. Keep them short and in plain language: the spec uses each step's text verbatim in `test.step`, so the
@@ -97,8 +115,13 @@ report and the trace read like the case.
97
115
  generated: `/gen-spec` stops with `needs-input`.
98
116
  - **Open questions** go in a `🚫 TODO [blocking]` or `❓ TODO [non-blocking]` block under the sentence they are about,
99
117
  with the agent's suggestion written above and a last line `- ✍️ Answer:`. Accepting a question means deleting the
100
- whole block; answering means writing after `Answer:`. The marker `🔍 to be confirmed by QA` / `👁️ not observed`
101
- marks something seen but not settled.
118
+ whole block; answering means writing after `Answer:`. Hang the block under the sentence that makes the claim, not
119
+ under a broader one above it, and one claim gets one block.
120
+ - **What the AI saw** is marked by what happens if it is wrong: `(seen on <profile>)` is a plain note the run checks by
121
+ itself, and `(observed on <profile>, 🔍 to be confirmed: <what a person must decide>)` is a line that would pass even
122
+ if it were wrong, so a person settles it. A marker may name a screenshot (`evidence: .explore/.../x.png`) so nobody
123
+ has to open the page again. Never put a marker inside a TODO block, on the sentence a block hangs under, or inside
124
+ the `## Values` yaml: one claim carries one signal. `pnpm case:check` fails on the first two.
102
125
 
103
126
  ## Expected
104
127
 
@@ -174,6 +197,7 @@ It reads every case (or the ids you pass) and prints one line per case. It **fai
174
197
  - a target card without its required fields (`What`, `Where`, `Identified by`), or a reference image outside
175
198
  `.explore/targets/<case-id>/`;
176
199
  - a `[T<n>]` reference with no card;
200
+ - a `## Rules` bullet that names `Step <n>`, `Before step <n>` or `After step <n>` when the case has no such step;
177
201
  - an invalid `## Values` block (bad YAML, an unknown profile, a non-map entry).
178
202
 
179
203
  It **warns** (usable, but something to settle) on: a reference image missing on this machine (`.explore/` is
package/docs/commands.md CHANGED
@@ -21,6 +21,7 @@ thin wrappers, so `pnpm asrun`, `pnpm signin` and the rest work as below. The sa
21
21
  | Read what the last report says about a case | `pnpm heal:evidence --case <case-id> --env <env>` |
22
22
  | List or forget test data that could not be removed | `pnpm cleanup --env <env> [--forget]` |
23
23
  | Create or validate a case | `pnpm case:new <id>`, `pnpm case:check [case-id]` |
24
+ | Settle the "to be confirmed" lines of a case from a passing run | `pnpm markers:confirm --case <case-id> --env <env> [--dry-run]` |
24
25
  | Check a spec | `pnpm spec:lint [case-id]`, `pnpm baselines:review [case-id] [--new]` |
25
26
  | Look at a PDF | `pnpm pdf:render <file.pdf> --out <dir> [--pages all] [--corner top-left]` |
26
27
  | Type-check the workspace | `pnpm typecheck` |
@@ -125,7 +126,11 @@ pnpm pdf:render report.pdf --out .explore/targets/header-logo --corner top-left
125
126
  ```
126
127
 
127
128
  - `case:new` creates the case from the template. One optional group folder, created on demand.
128
- - `case:check` validates front matter, values and target cards; see [cases.md](./cases.md).
129
+ - `case:check` validates front matter, values and target cards; it fails on a marker inside a question block, a marker
130
+ on the line a question hangs under, and a marker inside the `## Values` yaml. See [cases.md](./cases.md).
131
+ - `markers:confirm` reads the newest run report (or `--report <run-id|file>`) and removes the `🔍 to be confirmed`
132
+ markers whose line the spec carries as `// unconfirmed: ...`, but only when every test of the case passed. It answers
133
+ "does the page still say this?", never "is this the right expectation".
129
134
  - `spec:lint` is a static check of the specs: it fails on sleeps, `networkidle`, image checks without a target,
130
135
  `.first()` on images, raw download events, a magic-number-only PDF check. `pnpm spec:lint <case-id>` limits it.
131
136
  - `baselines:review` opens nothing; it lists every baseline next to the target card that says what it must show.
@@ -257,6 +257,29 @@
257
257
  .side .item { padding: 8px 16px 8px 12px; border-bottom: 1px solid var(--line); border-left: 4px solid transparent; font-family: var(--mono); font-size: 14px; }
258
258
  .side .item.selected { background: var(--theme-bg); border-left-color: var(--theme); }
259
259
 
260
+ /* Filter bar (mirror components/filter-chips.tsx and components/search-field.tsx). */
261
+ .filters { display: flex; flex-wrap: wrap; align-items: center; gap: 4px; }
262
+ .filters .group-label { margin-right: 2px; font-size: 12px; color: var(--ink-3); }
263
+ .filter-chip {
264
+ display: inline-flex; align-items: center; gap: 4px; height: 24px; padding: 0 6px; border-radius: 2px;
265
+ background: var(--card); color: var(--ink-2); font-size: 12px; box-shadow: inset 0 0 0 1px var(--line-mid);
266
+ }
267
+ .filter-chip .count { color: var(--ink-3); }
268
+ .filter-chip.on {
269
+ background: var(--theme-btn); color: #fff; font-weight: 500;
270
+ box-shadow: inset 0 0 0 1px var(--theme-btn);
271
+ }
272
+ .filter-chip.on .count { color: rgb(255 255 255 / .75); }
273
+ .filter-chip.on.production { background: var(--jiang); box-shadow: inset 0 0 0 1px var(--jiang); }
274
+ .filter-chip.on.staging { background: var(--cheng); box-shadow: inset 0 0 0 1px var(--cheng); }
275
+ .filter-chip.on.other { background: var(--gold-deep); box-shadow: inset 0 0 0 1px var(--gold-deep); }
276
+ .filter-search { position: relative; width: 260px; max-width: 100%; }
277
+ .filter-search svg { position: absolute; left: 8px; top: 50%; transform: translateY(-50%); color: var(--ink-3); }
278
+ .filter-search .fake {
279
+ display: flex; align-items: center; height: 32px; padding: 0 8px 0 28px; border: 1px solid var(--line-mid);
280
+ border-radius: 4px; background: #fff; color: var(--ink-3); font-size: 14px;
281
+ }
282
+
260
283
  .seal-stamp {
261
284
  display: inline-flex; align-items: center; justify-content: center; width: 28px; height: 28px; border-radius: 3px;
262
285
  background: var(--seal); color: var(--paper); font-family: var(--serif); font-weight: 400; font-size: 16px; line-height: 1;
@@ -302,6 +325,7 @@
302
325
  <a href="#badges">Badges</a>
303
326
  <a href="#action-colors">Action colors</a>
304
327
  <a href="#dropdowns">Dropdowns</a>
328
+ <a href="#filter-bars">Filter bars</a>
305
329
  <a href="#multi-value">Multi-value fields</a>
306
330
  <a href="#icons">Icons</a>
307
331
  <a href="#tooltips">Tooltips</a>
@@ -503,6 +527,7 @@
503
527
  <tr><td><code>info</code> (light blue)</td><td>A view toggle that must be easy to find, not an action on data: "Full screen" / "Restore" in the Jobs drawer bar. Blue = read-only.</td></tr>
504
528
  <tr><td><code>secondary</code> (white)</td><td>Everything else: Verify, Visit, Discard, Copy, navigation.</td></tr>
505
529
  <tr><td><code>danger</code> (white, red text)</td><td>Cancel a running job, destructive confirmations.</td></tr>
530
+ <tr><td><code>invalid</code> (red field)</td><td>A control that is required and still empty, drawn as a button because it opens a panel instead of a native list (the environment picker of the case toolbar). The same red as an invalid dropdown; it is not an action.</td></tr>
506
531
  </tbody>
507
532
  </table>
508
533
  <p class="small muted">Hover lifts the button 1px; disabled is 50% opacity with no shadow. With an icon, add <code>gap-1.5</code> to the button.</p>
@@ -522,7 +547,7 @@
522
547
  <tbody>
523
548
  <tr><td><span class="badge green">green</span></td><td>ok: Logged in</td><td>Succeeded</td></tr>
524
549
  <tr><td><span class="badge red">red</span></td><td>expired, missing</td><td>Failed</td></tr>
525
- <tr><td><span class="badge amber">amber</span></td><td>unverified, error</td><td>Needs your input (an AI run that ended with <code>RESULT: needs-input</code>)</td></tr>
550
+ <tr><td><span class="badge amber">amber</span></td><td>unverified, error</td><td>Needs your input (an AI run that ended with <code>RESULT: needs-input</code>) or No result (a run that printed no <code>RESULT:</code> line at all)</td></tr>
526
551
  <tr><td><span class="badge blue">blue</span></td><td>-</td><td>Running</td></tr>
527
552
  <tr><td><span class="badge gray">gray</span></td><td>-</td><td>Cancelled</td></tr>
528
553
  </tbody>
@@ -584,7 +609,7 @@
584
609
  <tbody>
585
610
  <tr><td><code>explore-case</code></td><td>blue (qing)</td><td>Edits only the case file (wording, observations, open questions, inputs). The agent only reads pages; you do the steps that write data.</td></tr>
586
611
  <tr><td><code>gen-spec</code></td><td>amber (gold)</td><td>Writes: creates the spec and fills the case's <code>## Values</code>. Does not run the tests.</td></tr>
587
- <tr><td><code>heal-spec</code></td><td>red (seal)</td><td>Rewrites an existing spec and runs the tests.</td></tr>
612
+ <tr><td><code>heal-spec</code></td><td>red (seal)</td><td>Repairs the spec or refines the case and the spec, then runs the tests.</td></tr>
588
613
  </tbody>
589
614
  </table>
590
615
  <p class="small muted">
@@ -600,6 +625,14 @@
600
625
  <code>Select</code> is a button that draws its own list under it, in a <code>card-popup</code> panel, which is what
601
626
  gives every dropdown the same raised look.
602
627
  </p>
628
+ <p class="small muted">
629
+ Two controls put their label inside the button and open a panel with a title rather than a bare list: the
630
+ environment picker of the case toolbar (<code>Env: staging</code>, <code>components/env-picker.tsx</code>) and the
631
+ profile picker beside it (<code>Profiles: 2 of 3</code>, <code>components/profile-picker.tsx</code>). They are
632
+ buttons because their list needs a title and a line of explanation, not a <code>Select</code>; they keep this
633
+ section's height, <code>card-popup</code> panel, keyboard and click-outside rules. Do not turn them back into a
634
+ <code>Select</code> that repeats the label on every row.
635
+ </p>
603
636
  <div class="demo">
604
637
  <span class="select"><span class="fake">staging</span><svg class="chev" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="6 9 12 15 18 9"></polyline></svg></span>
605
638
  <span class="select focus"><span class="fake">staging</span><svg class="chev" xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="6 9 12 15 18 9"></polyline></svg></span>
@@ -654,6 +687,60 @@
654
687
  </tbody>
655
688
  </table>
656
689
 
690
+ <h3 id="filter-bars">Filter bars</h3>
691
+ <p>
692
+ A list that can get long (the profiles of the workspace, the profiles of a case's Values table) gets a filter bar
693
+ above it: <code>SearchField</code> and <code>FilterChips</code> from <code>ui/web/src/components/</code>. They are
694
+ controls only; the page decides what a value means and filters its own rows.
695
+ </p>
696
+ <div class="demo" style="display:block">
697
+ <div class="filter-search" style="margin-bottom:8px">
698
+ <svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="11" cy="11" r="8"></circle><line x1="21" y1="21" x2="16.65" y2="16.65"></line></svg>
699
+ <span class="fake">atpil</span>
700
+ </div>
701
+ <div style="display:flex;flex-wrap:wrap;gap:8px 16px">
702
+ <div class="filters">
703
+ <span class="group-label">Env</span>
704
+ <span class="filter-chip">All <span class="count">17</span></span>
705
+ <span class="filter-chip on production">PRODUCTION <span class="count">7</span></span>
706
+ <span class="filter-chip on staging">STAGING <span class="count">8</span></span>
707
+ <span class="filter-chip">SANDBOX <span class="count">2</span></span>
708
+ </div>
709
+ <div class="filters">
710
+ <span class="group-label">Tag</span>
711
+ <span class="filter-chip">All <span class="count">17</span></span>
712
+ <span class="filter-chip on">atp <span class="count">6</span></span>
713
+ <span class="filter-chip">compass <span class="count">4</span></span>
714
+ <span class="filter-chip">corp <span class="count">4</span></span>
715
+ </div>
716
+ </div>
717
+ <p class="small muted" style="margin:8px 0 0">Showing 6 of 17 profiles · <span style="text-decoration:underline">Clear filters</span></p>
718
+ </div>
719
+ <p class="small muted">
720
+ Search "atpil" with the dashes left out, both environments pressed and one tag: the two groups are combined with
721
+ AND.
722
+ </p>
723
+ <pre class="card mono" style="overflow:auto">import {SearchField} from '../components/search-field';
724
+ import {FilterChips} from '../components/filter-chips';
725
+
726
+ &lt;SearchField value={query} onChange={setQuery} ariaLabel="Search profiles" placeholder="Search id, name..." /&gt;
727
+ &lt;FilterChips label="Env" selected={envFilter} onChange={setEnvFilter} allCount={profiles.length}
728
+ options={envs.map((env) =&gt; ({value: env, label: env.toUpperCase(), count: countOf(env), tone: envFillClass(env)}))} /&gt;</pre>
729
+ <table>
730
+ <thead><tr><th>Part</th><th>Rule</th></tr></thead>
731
+ <tbody>
732
+ <tr><td>Search</td><td>One <code>SearchField</code> at the top. It looks in id, name, account, environment, tags and address; every word of the query must match one of them. A compact copy without <code>-</code>, <code>_</code>, <code>.</code> or spaces lets <code>atpil</code> find <code>atp-il-staging</code>.</td></tr>
733
+ <tr><td>Groups</td><td>One <code>FilterChips</code> per kind of value (environment, tag). Inside a group a row matches <b>any</b> pressed value; the groups are combined with <b>AND</b>. An empty group filters nothing.</td></tr>
734
+ <tr><td>All</td><td>The first chip of a group clears that group only. It is pressed while the group has no selection, so "no filter" can always be told from "everything selected".</td></tr>
735
+ <tr><td>Counts</td><td>Each option shows how many rows it would leave, and <code>All</code> shows the group's total. Counts come from the unfiltered list, so they do not dance while a value is pressed.</td></tr>
736
+ <tr><td>Pressed look</td><td>Neutral chips take the page accent (<code>theme-btn</code>, white text). Environment chips pass <code>envFillClass(env)</code>, the same solid marker as <code>EnvBadge</code>, so color means the environment here too.</td></tr>
737
+ <tr><td>State</td><td>The chip is the button: <code>aria-pressed</code>, a tooltip with its name (and "pressed" while on), and the count as a second, quieter part of the label.</td></tr>
738
+ <tr><td>Feedback</td><td>While a filter is active, show <code>Showing N of M</code> and <b>Clear filters</b>. When nothing matches, the empty state repeats the clear action instead of leaving a blank table.</td></tr>
739
+ <tr><td>Not remembered</td><td>A filter is a way of looking, not a setting: do not put it in <code>localStorage</code>. A reload shows everything, so nothing looks missing after lunch.</td></tr>
740
+ <tr><td>In the case panel</td><td>The Values table adds a two-button <b>Rows: Keys / Profiles</b> switch beside the search as soon as a case names two profiles, and shows the search and the environment chips only when it names more than three. The filters and the switch belong to that case; opening another case starts unfiltered.</td></tr>
741
+ </tbody>
742
+ </table>
743
+
657
744
  <h2 id="multi-value">Multi-value fields</h2>
658
745
  <p>
659
746
  A field whose value is a list is never a <a href="#dropdowns">Dropdown</a> plus an "Add" text input: that mixes a
@@ -829,11 +916,12 @@
829
916
  </p>
830
917
  <p class="muted">
831
918
  The preview block that holds the editor caret is shaded with <code>theme-bg</code> at 60% and a 2px <code>theme</code> left
832
- border (an overlay, so it follows the page theme). The editor and the preview scroll together and a click in the preview
833
- moves the caret; the logic is in <code>ui/web/src/lib/preview-sync.ts</code>.
919
+ border (an overlay, so it follows the page theme). The overlay paints <b>behind</b> the text, never over it: the
920
+ highlighted block keeps its own text color instead of looking washed out. The editor and the preview scroll together
921
+ and a click in the preview moves the caret; the logic is in <code>ui/web/src/lib/preview-sync.ts</code>.
834
922
  </p>
835
923
  <p class="muted">
836
- The previous / next buttons of the case toolbar move the caret to the next marker and blink that marker for a moment in
924
+ The previous / next buttons at the right end of the editor toolbar move the caret to the next marker and blink that marker for a moment in
837
925
  both panes (<code>cm-kw-flash</code> in the editor, <code>kw-flash</code> over the <code>&lt;mark&gt;</code> of the
838
926
  document): <code>gold-glow</code> at 65% behind it and a 2px <code>dai-deep</code> ring with a wider
839
927
  <code>gold-glow</code> glow around it, the ring blinking on and off three times (0.45&nbsp;s each), then it goes away.
@@ -57,6 +57,11 @@ and regenerate the spec.
57
57
  `Identified by`), refers to a reference image outside `.explore/targets/<case-id>/`, or a check refers to a `[T<n>]`
58
58
  that has no card. A blocking question (`🚫 TODO [blocking]`) is the normal way to wait for a person's answer.
59
59
 
60
+ **`pnpm case:check` fails on a marker** — the same claim carries two signals: a marker word inside a question block, a
61
+ marker on the line a question hangs under, or a marker inside the `## Values` yaml (where it would become part of the
62
+ expected value). Keep one of the two, and move a `## Values` marker to the prose line above the block. A marker without
63
+ a reason is only a warning: write `🔍 to be confirmed: <what a person must decide>`.
64
+
60
65
  **`pnpm spec:lint` fails** — read the rule name in the message: `no-sleep` (`waitForTimeout`), `no-networkidle`,
61
66
  download handling, a screenshot or image check without `// target: T<n>`, a PDF check on a magic number alone. The
62
67
  fixes are in the framework's `generating-specs` conventions; do not silence the rule.
@@ -69,6 +74,10 @@ is recorded.
69
74
  checkout has the card but not the screenshot. Run the exploration again on this machine (or look at the card and the
70
75
  report's baseline).
71
76
 
77
+ **`/explore-case` takes long with many profiles** — pass `--profiles a,b` (the toolbar's profile picker does the same):
78
+ only those profiles are opened, and a later run with the others adds what it finds. `/gen-spec` has nothing to pick: it
79
+ covers every profile the case names.
80
+
72
81
  ## Installing
73
82
 
74
83
  **`pnpm add` installs the previous release instead of `latest`** — pnpm holds back versions published in the last
@@ -113,6 +122,41 @@ cases, specs, `profiles.yaml` and `package.json` stay. Then run `pnpm dlx @asrun
113
122
  `claude mcp list`; `failed: ... path specified` means the `playwright` path in that file does not resolve (repair it
114
123
  with `pnpm asrun init --force <file>`).
115
124
 
125
+ **A `browser_*` tool says `Tool "<name>" not found`** — expected for part of them: the `playwright-test` MCP
126
+ (`run-test-mcp-server`) lists every browser tool Playwright has, but a call runs in a paused test whose backend installs
127
+ only the `core` and `testing` capabilities, plus it never installs the `skillOnly` tools. The server is up and the other
128
+ tools work: use the substitute in `.claude/skills/exploring-cases/references/tools.md` and do not report a missing MCP
129
+ server. `browser_highlight` / `browser_hide_highlight` become `locator.highlight()` / `page.hideHighlight()` through
130
+ `browser_run_code_unsafe`, `browser_network_clear` becomes a marker on the request list,
131
+ `browser_start_recording` / `browser_stop_recording` are never called (the handoff card,
132
+ `.claude/skills/exploring-cases/references/handoff-card.js`, starts and stops the recording itself and returns
133
+ `recorded`). `browser_annotate` cannot run either: the person names the element in the handoff card. `pnpm asrun doctor`
134
+ prints how many tools the installed Playwright runs and warns when `tools.md` names one it cannot.
135
+
136
+ **An AI run stops answering and never finishes** — the log keeps repeating `... still waiting for <tool call> (running
137
+ 5m 30s)` or `... no output from the agent for 5m`, and nothing else arrives. A workspace has exactly one
138
+ `playwright-test` MCP server (the `run-test-mcp-server` child process OpenCode starts from `opencode.jsonc`), shared by
139
+ every session in that workspace, and Playwright runs one paused test in it at a time. One interrupted browser call —
140
+ closing the browser window by hand, cancelling a run while a call is in flight, or a second AI run using the same
141
+ server — leaves that queue wedged: the call never returns and every later run hangs on its first
142
+ `generator_setup_page` until the process is replaced. `pnpm asrun doctor` cannot see it, because it starts a fresh
143
+ server of its own. Cancel the stuck run, quit and restart OpenCode (that ends its MCP processes), then start the run
144
+ again. From the UI this is bounded: only one AI run at a time runs per workspace (the `ai` job lock, whatever case it is
145
+ on), and a run that says nothing for 10 minutes — 15 while a handoff card waits for a person — is stopped with this
146
+ instruction in its log.
147
+
148
+ **The UI server restarted while an AI run was going** — the job is still in the Jobs drawer, marked **Interrupted**,
149
+ and its log stops where the server died. If the agent (and the `playwright-test` MCP server below it) is still running,
150
+ the job offers **Stop the leftover process**; stop it before starting another AI run, or both agents share one MCP
151
+ server and wedge it. A job that had already written `RESULT: needs-input` is recovered as **Succeeded**, so its answers
152
+ can still be sent. Every finished job has **Run again**, which reuses the inputs it was started with.
153
+
154
+ **A call fails with `Missing key`** — the published schema requires two things that are easy to leave out: `intent` on
155
+ every action, input and assertion tool (`browser_click`, `browser_type`, `browser_evaluate`, `browser_run_code_unsafe`,
156
+ `browser_wait_for`, ...), and the fields that have a default (`browser_take_screenshot` needs `scale: "css"`,
157
+ `browser_network_requests` needs `static: false`, `generator_setup_page` needs `plan`). The full list is in
158
+ `.claude/skills/exploring-cases/references/tools.md`.
159
+
116
160
  **An AI job runs but uses a slow or limited model** — the top bar's **Model** picker sets the model for UI runs;
117
161
  without one, a one-shot OpenCode run uses its built-in free model, which has a daily usage limit. Set `"model"` in
118
162
  `opencode.jsonc` to stop depending on the browser's choice. For Claude Code, the picker lists what its settings,
@@ -131,6 +175,6 @@ read it. `doctor` lists the overrides it sees.
131
175
  project. Add `--reset-rules` to replace them with the template, and expect to lose the project's rules.
132
176
 
133
177
  **The AI did something you did not expect** — exploring and generating only read pages and edit the case file; healing
134
- edits the spec and runs it under the same write gate as any run (staging writes are allowed, any other environment
135
- needs a person's confirmation). None of them writes product data on its own, and none updates a baseline. If a step
136
- needs a person to write, run it with `--attended`; that is where a person acts, in the browser.
178
+ edits the case and the spec and runs the spec under the same write gate as any run (staging writes are allowed, any
179
+ other environment needs a person's confirmation). None of them writes product data on its own, and none updates a
180
+ baseline. If a step needs a person to write, run it with `--attended`; that is where a person acts, in the browser.