@asrunjs/asrun 0.2.0 → 0.5.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (94) hide show
  1. package/.claude/commands/explore-case.md +18 -15
  2. package/.claude/commands/gen-spec.md +16 -13
  3. package/.claude/commands/heal-spec.md +26 -23
  4. package/.claude/commands/make-case.md +11 -8
  5. package/.claude/skills/exploring-cases/SKILL.md +146 -132
  6. package/.claude/skills/exploring-cases/references/open-questions.md +116 -112
  7. package/.claude/skills/exploring-cases/references/readiness.md +146 -137
  8. package/.claude/skills/exploring-cases/references/user-handoff.md +157 -147
  9. package/.claude/skills/generating-specs/SKILL.md +219 -207
  10. package/.claude/skills/generating-specs/references/conventions.md +148 -148
  11. package/.claude/skills/making-cases/SKILL.md +72 -64
  12. package/.claude/skills/making-cases/references/normalizing.md +39 -38
  13. package/.claude/skills/making-cases/references/writes-and-email.md +37 -36
  14. package/AGENTS.md +159 -154
  15. package/CLAUDE.md +2 -1
  16. package/README.md +512 -410
  17. package/dist/lib/build-artifacts.d.ts +8 -0
  18. package/dist/lib/build-artifacts.js +10 -0
  19. package/dist/lib/case-doc.d.ts +311 -0
  20. package/dist/lib/case-doc.js +1117 -0
  21. package/dist/lib/cases.d.ts +2 -15
  22. package/dist/lib/cases.js +5 -78
  23. package/dist/lib/command.d.ts +24 -0
  24. package/dist/lib/command.js +90 -0
  25. package/dist/lib/fixtures.js +2 -2
  26. package/dist/lib/login-card.d.ts +22 -0
  27. package/dist/lib/login-card.js +183 -0
  28. package/dist/lib/node-cli.d.ts +6 -2
  29. package/dist/lib/node-cli.js +16 -6
  30. package/dist/lib/profiles.d.ts +25 -1
  31. package/dist/lib/profiles.js +54 -1
  32. package/dist/lib/run-filters.js +10 -2
  33. package/dist/lib/session.d.ts +13 -1
  34. package/dist/lib/session.js +73 -7
  35. package/dist/lib/targets.d.ts +5 -13
  36. package/dist/lib/targets.js +4 -29
  37. package/dist/lib/values.js +2 -2
  38. package/dist/scripts/build.js +2 -1
  39. package/dist/scripts/check-cases.js +74 -6
  40. package/dist/scripts/doctor.js +115 -4
  41. package/dist/scripts/explore.js +22 -1
  42. package/dist/scripts/init.d.ts +7 -1
  43. package/dist/scripts/init.js +61 -13
  44. package/dist/scripts/lint-specs.js +4 -1
  45. package/dist/scripts/login.js +30 -2
  46. package/dist/ui/server/actions.d.ts +4 -0
  47. package/dist/ui/server/actions.js +25 -12
  48. package/dist/ui/server/ai-models.d.ts +3 -0
  49. package/dist/ui/server/ai-models.js +347 -0
  50. package/dist/ui/server/ai-stream.d.ts +34 -9
  51. package/dist/ui/server/ai-stream.js +283 -77
  52. package/dist/ui/server/app.js +74 -12
  53. package/dist/ui/server/jobs.d.ts +7 -1
  54. package/dist/ui/server/jobs.js +69 -15
  55. package/dist/ui/server/profiles.d.ts +20 -1
  56. package/dist/ui/server/profiles.js +214 -38
  57. package/dist/ui/shared/ai-commands.d.ts +16 -6
  58. package/dist/ui/shared/ai-commands.js +35 -12
  59. package/dist/ui/shared/types.d.ts +114 -0
  60. package/docs/agents/framework-dev.md +79 -0
  61. package/docs/theme-guide.html +922 -692
  62. package/package.json +12 -1
  63. package/templates/.claude/commands/explore-case.md +22 -21
  64. package/templates/.claude/commands/gen-spec.md +21 -20
  65. package/templates/.claude/commands/heal-spec.md +31 -30
  66. package/templates/.claude/commands/make-case.md +13 -12
  67. package/templates/.mcp.json +8 -8
  68. package/templates/AGENTS.md +64 -45
  69. package/templates/cases/_template.md +57 -63
  70. package/templates/opencode.jsonc +16 -11
  71. package/templates/profiles.yaml +42 -29
  72. package/ui/dist/assets/{dist-BHaIOZdW.js → dist-BGbe0-Aa.js} +1 -1
  73. package/ui/dist/assets/{dist-CBV7_vDN.js → dist-Ba-HC662.js} +1 -1
  74. package/ui/dist/assets/{dist-BiE4suo9.js → dist-BtMLMthJ.js} +1 -1
  75. package/ui/dist/assets/{dist-gF2bFn-5.js → dist-C-Oyo1ZT.js} +1 -1
  76. package/ui/dist/assets/{dist-DX8knLVr.js → dist-C8MdhqgG.js} +1 -1
  77. package/ui/dist/assets/{dist-DHCY9KpV.js → dist-CG1pHn5m.js} +1 -1
  78. package/ui/dist/assets/{dist-BwYRbe5s.js → dist-CmpteMYY.js} +1 -1
  79. package/ui/dist/assets/{dist-D_I45xgJ.js → dist-DSy3dVf4.js} +1 -1
  80. package/ui/dist/assets/{dist-BqAROxzI.js → dist-Dq6sW5Ur.js} +1 -1
  81. package/ui/dist/assets/{dist-BSJMM_jp.js → dist-jjxQutEm.js} +1 -1
  82. package/ui/dist/assets/index-CTmnF0Wp.css +2 -0
  83. package/ui/dist/assets/index-P_Yo0ZZn.js +338 -0
  84. package/ui/dist/help/ai-tools.png +0 -0
  85. package/ui/dist/help/case-explore.png +0 -0
  86. package/ui/dist/help/case-generate.png +0 -0
  87. package/ui/dist/help/cases-memo.png +0 -0
  88. package/ui/dist/help/profile-form.png +0 -0
  89. package/ui/dist/help/profiles-login.png +0 -0
  90. package/ui/dist/help/reports.png +0 -0
  91. package/ui/dist/help/run-tests.png +0 -0
  92. package/ui/dist/index.html +22 -16
  93. package/ui/dist/assets/index-DmasQ6Rq.css +0 -2
  94. package/ui/dist/assets/index-I7uU93kk.js +0 -294
@@ -1,112 +1,116 @@
1
- # Open questions: write, resolve, report
2
-
3
- Shared by `exploring-cases` and `generating-specs`. The agent never waits for an answer in the chat: it may run
4
- without anybody present (a one-shot `claude -p` / `opencode run`). The case file is where the agent and the user
5
- talk to each other, so nothing is lost between runs, tools or sessions.
6
-
7
- ## How a question is written in the case file
8
- Every marker starts with its emoji, which is part of the marker: `🚫 TODO [blocking]`, `❓ TODO [non-blocking]`, `✍️ Answer:`, `🔍 to be confirmed by QA` and `👁️ not observed`. The web UI colors the emoji with the marker and offers an X button that deletes it.
9
- Every question is a small block that hangs under a normal sentence. The sentence is the agent's suggestion, written
10
- exactly as it should read in the final case. The block holds the question and an empty answer slot:
11
-
12
- ```
13
- - <the suggested statement, a complete sentence that can stand in the case as it is>
14
- - ❓ TODO [non-blocking]: <the question, one sentence>? (<evidence, e.g. seen on example-staging>)
15
- - ✍️ Answer:
16
- ```
17
-
18
- The same shape is used under a step, under a bullet of "Expected", under the yaml block of "Values" and in "Data":
19
-
20
- ```
21
- 11. Delete the document with the "Delete" option of its "..." menu on the list page. [writes]
22
- - 🚫 TODO [blocking]: How is the document deleted? (the same path was observed for drafts on example-staging)
23
- - ✍️ Answer:
24
- ```
25
-
26
- Rules for writing it:
27
- - **The suggestion is a statement, never a question or a half sentence.** "The logo is visible and not empty." is
28
- right. "Suggested: visible." and "what is the logo?" are wrong. When no value is known, state the safest reading
29
- (visible and not empty, the spec accepts both, no screenshot baseline is taken, ...). Never invent a concrete
30
- value such as a text, a count or a color.
31
- - **The suggestion is always the parent line**, so deleting the TODO block leaves a finished case. Do not put the
32
- suggestion inside the TODO line, and do not write `Suggested:`.
33
- - **Always leave the empty `Answer:` line** as the last line of the block, one level deeper than the `TODO` line.
34
- - The TODO line has the question and, when the page showed something, the evidence in parentheses. Nothing else.
35
- - `[blocking]`: the spec cannot be written correctly without the answer. `[non-blocking]`: the suggestion is safe;
36
- QA should confirm it later but nothing waits for it.
37
- - One question per block; narrow broad questions down instead of copying them ("which profile elements?" becomes
38
- "is the logo the only per-profile element in the left side bar?").
39
- - A value read from a page is not a question: write it as a statement marked `(observed, 🔍 to be confirmed by QA)`.
40
- It needs no TODO block; the reviewer confirms it by deleting the marker.
41
-
42
- ## How the user answers
43
- Two ways, and the agent must understand both on the next run:
44
- - **The suggestion is fine:** delete the whole TODO block (the `TODO` line and the `Answer:` line). The parent
45
- sentence is now confirmed text.
46
- - **Own answer:** write it after `Answer:`. The answer replaces the parent sentence where they disagree. The user
47
- may delete the rest of the block, or leave it.
48
- - An `Answer:` with no text is not an answer. A block that was left untouched stays open.
49
- - An old `TODO: ...` line without a tag or without a parent suggestion (written before this format) counts as
50
- `[blocking]` until it is rewritten into the shape above.
51
-
52
- The agent folds an answer into plain text in the right place (the parent sentence, an expected bullet, an entry of
53
- `## Values`, a step) and removes the `TODO` line and the `Answer:` line. If an answer contradicts the page, it keeps the
54
- answer, adds `(page shows <x> on <profile-id>)` and reports it as a finding. It never overrides the user.
55
-
56
- ## What an open block means for the agent
57
- - No TODO block under a sentence: the sentence is confirmed. Use it.
58
- - `[non-blocking]` block: use the parent sentence as the expectation; keep the block for QA.
59
- - `[blocking]` block: the parent sentence is only a guess. Do not generate a check from it: write
60
- `test.fixme(true, '<the question>')` inside the step (or skip that profile).
61
- - Every run re-evaluates the blocks that are already in the file against the rules below, including their tag: a
62
- `[blocking]` question that now has an observation or a safe default is downgraded or closed in the file, and the
63
- edit is part of the report. Do not carry old tags over.
64
-
65
- ## Resolution order (no question goes to the user while a better step exists)
66
- 1. **The page answers it.** Facts such as exact labels, which controls exist, where the logo is, what a menu holds
67
- are answered by exploration (`/explore-case`; needs an environment and a logged-in profile; the agent only reads, a person does the steps that write). Rewrite the parent sentence
68
- with the fact, mark it `(observed, 🔍 to be confirmed by QA)` and remove the block.
69
- - A check with no expected value ("check X", "can see Y") becomes: the observed behaviour, marked as observed.
70
- A spec may be generated from it.
71
- - Values that a check asserts (logo name, label, option list) are observed per profile and written into the case's
72
- `## Values`, under the profile id (or `default` when they hold for every profile).
73
- 2. **A default is safe.** Use the default of `gaps.md` in the `making-cases` skill (visible and not empty, all
74
- environments, `suites: [regression]`, ...) as the parent sentence and keep the block as `[non-blocking]`.
75
- Implementation questions are in this group: whether "visible" is enough, how long to wait, which of two equally
76
- good locators for the SAME element to use. Pick the most robust option the page offers, state it, and do not
77
- block on it. **Not in this group: which element a check looks at.** That is a target (`assertion-targets.md`):
78
- one clear candidate becomes a card, anything else blocks.
79
- 3. **Otherwise it blocks.** Keep the block as `[blocking]`, narrowed as far as the page allows, with the best guess
80
- as the parent sentence.
81
-
82
- ## What is always blocking (never guess, never generate around it)
83
- - A step that would write data or send email while the front matter has no `writes` list (or no `writes-note`), or for an environment that `writes` does not list.
84
- - A step that writes data or needs a choice, in a run that is not attended: no person can do it, so there is nothing to
85
- record under `## Inputs`.
86
- - A step that cannot be performed after exploring: the path 404s, the control does not exist for any profile, or it
87
- needs data that does not exist and the case does not create it.
88
- - Observed behaviour that contradicts what the case states, or a profile that deviates from the others where the case
89
- says "same for every profile". This may be a product bug: record it (`(observed: <what>, on <profile-id>)`),
90
- do not encode the deviating value as the expectation, and say it in the report.
91
- - **An assertion target that is in doubt** (`assertion-targets.md`): several candidates, only a position or a generic
92
- locator to find it, dynamic data in its area, no stable identity, or the page differs from the case. The question
93
- names the exact image, text, place or UI wanted and lists the candidates. Never a safe default, because a wrong
94
- guess passes silently.
95
- - An intent question the page cannot answer ("should this also work for agents?").
96
- - No environment was given, or no profile is logged in, so a fact that needs the page cannot be observed.
97
-
98
- ## Final report
99
- Report in the user's language. List every open block with its question and the parent sentence it would confirm, so
100
- the user can see what deleting it accepts. The last line of the final message is exactly one of:
101
-
102
- ```
103
- RESULT: ready
104
- RESULT: needs-input
105
- ```
106
-
107
- - `ready`: no `[blocking]` question remains. Non-blocking ones may remain; list them so QA can confirm them.
108
- - `needs-input`: at least one `[blocking]` question remains in the case file (list each with its parent sentence),
109
- or the work could not be done (no environment, nobody logged in). Say what the user should edit in the case file,
110
- then re-run.
111
-
112
- Do not write anything after the `RESULT:` line.
1
+ # Open questions: write, resolve, report
2
+
3
+ Shared by `exploring-cases` and `generating-specs`. The agent never waits for an answer in the chat: it may run
4
+ without anybody present (a one-shot `claude -p` / `opencode run`). The case file is where the agent and the user
5
+ talk to each other, so nothing is lost between runs, tools or sessions.
6
+
7
+ ## How a question is written in the case file
8
+ Every marker starts with its emoji, which is part of the marker: `🚫 TODO [blocking]`, `❓ TODO [non-blocking]`, `✍️ Answer:`, `🔍 to be confirmed by QA` and `👁️ not observed`. The web UI colors the emoji with the marker and offers an X button that deletes it.
9
+ Every question is a small block that hangs under a normal sentence. The sentence is the agent's suggestion, written
10
+ exactly as it should read in the final case. The block holds the question and an empty answer slot:
11
+
12
+ ```
13
+ - <the suggested statement, a complete sentence that can stand in the case as it is>
14
+ - ❓ TODO [non-blocking]: <the question, one sentence>? (<evidence, e.g. seen on example-staging>)
15
+ - ✍️ Answer:
16
+ ```
17
+
18
+ The same shape is used under a step, under a bullet of "Expected" and under the yaml block of "Values":
19
+
20
+ ```
21
+ 11. Delete the document with the "Delete" option of its "..." menu on the list page. [writes]
22
+ - 🚫 TODO [blocking]: How is the document deleted? (the same path was observed for drafts on example-staging)
23
+ - ✍️ Answer:
24
+ ```
25
+
26
+ Rules for writing it:
27
+ - **The suggestion is a statement, never a question or a half sentence.** "The logo is visible and not empty." is
28
+ right. "Suggested: visible." and "what is the logo?" are wrong. When no value is known, state the safest reading
29
+ (visible and not empty, the spec accepts both, no screenshot baseline is taken, ...). Never invent a concrete
30
+ value such as a text, a count or a color.
31
+ - **The suggestion is always the parent line**, so deleting the TODO block leaves a finished case. Do not put the
32
+ suggestion inside the TODO line, and do not write `Suggested:`.
33
+ - **Always leave the empty `Answer:` line** as the last line of the block, one level deeper than the `TODO` line.
34
+ Nothing goes after it, and there is exactly one per block.
35
+ - **Both the suggestion and the question may wrap.** A wrapped line is plain text (no bullet): the block ends at the
36
+ first blank line or at a line that is not indented deeper than the `TODO` line. Indent the block by 4 spaces under a
37
+ numbered step (2 under a `-` bullet) so Markdown renders it inside that step; `pnpm case:check` warns otherwise.
38
+ - The TODO line has the question and, when the page showed something, the evidence in parentheses. Nothing else.
39
+ - `[blocking]`: the spec cannot be written correctly without the answer. `[non-blocking]`: the suggestion is safe;
40
+ QA should confirm it later but nothing waits for it.
41
+ - One question per block; narrow broad questions down instead of copying them ("which profile elements?" becomes
42
+ "is the logo the only per-profile element in the left side bar?").
43
+ - A value read from a page is not a question: write it as a statement marked `(observed, 🔍 to be confirmed by QA)`.
44
+ It needs no TODO block; the reviewer confirms it by deleting the marker.
45
+
46
+ ## How the user answers
47
+ Two ways, and the agent must understand both on the next run:
48
+ - **The suggestion is fine:** delete the whole TODO block (the `TODO` line and the `Answer:` line). The parent
49
+ sentence is now confirmed text.
50
+ - **Own answer:** write it after `Answer:`. The answer replaces the parent sentence where they disagree. The user
51
+ may delete the rest of the block, or leave it.
52
+ - An `Answer:` with no text is not an answer. A block that was left untouched stays open.
53
+ - An old `TODO: ...` line without a tag or without a parent suggestion (written before this format) counts as
54
+ `[blocking]` until it is rewritten into the shape above.
55
+
56
+ The agent folds an answer into plain text in the right place (the parent sentence, an expected bullet, an entry of
57
+ `## Values`, a step) and removes the `TODO` line and the `Answer:` line. If an answer contradicts the page, it keeps the
58
+ answer, adds `(page shows <x> on <profile-id>)` and reports it as a finding. It never overrides the user.
59
+
60
+ ## What an open block means for the agent
61
+ - No TODO block under a sentence: the sentence is confirmed. Use it.
62
+ - `[non-blocking]` block: use the parent sentence as the expectation; keep the block for QA.
63
+ - `[blocking]` block: the parent sentence is only a guess. Do not generate a check from it: write
64
+ `test.fixme(true, '<the question>')` inside the step (or skip that profile).
65
+ - Every run re-evaluates the blocks that are already in the file against the rules below, including their tag: a
66
+ `[blocking]` question that now has an observation or a safe default is downgraded or closed in the file, and the
67
+ edit is part of the report. Do not carry old tags over.
68
+
69
+ ## Resolution order (no question goes to the user while a better step exists)
70
+ 1. **The page answers it.** Facts such as exact labels, which controls exist, where the logo is, what a menu holds
71
+ are answered by exploration (`/explore-case`; needs an environment and a logged-in profile; the agent only reads, a person does the steps that write). Rewrite the parent sentence
72
+ with the fact, mark it `(observed, 🔍 to be confirmed by QA)` and remove the block.
73
+ - A check with no expected value ("check X", "can see Y") becomes: the observed behaviour, marked as observed.
74
+ A spec may be generated from it.
75
+ - Values that a check asserts (logo name, label, option list) are observed per profile and written into the case's
76
+ `## Values`, under the profile id (or `default` when they hold for every profile).
77
+ 2. **A default is safe.** Use the default of `gaps.md` in the `making-cases` skill (visible and not empty, all
78
+ environments, `suites: [regression]`, ...) as the parent sentence and keep the block as `[non-blocking]`.
79
+ Implementation questions are in this group: whether "visible" is enough, how long to wait, which of two equally
80
+ good locators for the SAME element to use. Pick the most robust option the page offers, state it, and do not
81
+ block on it. **Not in this group: which element a check looks at.** That is a target (`assertion-targets.md`):
82
+ one clear candidate becomes a card, anything else blocks.
83
+ 3. **Otherwise it blocks.** Keep the block as `[blocking]`, narrowed as far as the page allows, with the best guess
84
+ as the parent sentence.
85
+
86
+ ## What is always blocking (never guess, never generate around it)
87
+ - A step that would write data or send email while the front matter has no `writes` list (or no `writes-note`), or for an environment that `writes` does not list.
88
+ - A step that writes data or needs a choice, in a run that is not attended: no person can do it, so there is nothing to
89
+ record as a `- Recipe:` line under it.
90
+ - A step that cannot be performed after exploring: the path 404s, the control does not exist for any profile, or it
91
+ needs data that does not exist and the case does not create it.
92
+ - Observed behaviour that contradicts what the case states, or a profile that deviates from the others where the case
93
+ says "same for every profile". This may be a product bug: record it (`(observed: <what>, on <profile-id>)`),
94
+ do not encode the deviating value as the expectation, and say it in the report.
95
+ - **An assertion target that is in doubt** (`assertion-targets.md`): several candidates, only a position or a generic
96
+ locator to find it, dynamic data in its area, no stable identity, or the page differs from the case. The question
97
+ names the exact image, text, place or UI wanted and lists the candidates. Never a safe default, because a wrong
98
+ guess passes silently.
99
+ - An intent question the page cannot answer ("should this also work for agents?").
100
+ - No environment was given, or no profile is logged in, so a fact that needs the page cannot be observed.
101
+
102
+ ## Final report
103
+ Report in the user's language. List every open block with its question and the parent sentence it would confirm, so
104
+ the user can see what deleting it accepts. The last line of the final message is exactly one of:
105
+
106
+ ```
107
+ RESULT: ready
108
+ RESULT: needs-input
109
+ ```
110
+
111
+ - `ready`: no `[blocking]` question remains. Non-blocking ones may remain; list them so QA can confirm them.
112
+ - `needs-input`: at least one `[blocking]` question remains in the case file (list each with its parent sentence),
113
+ or the work could not be done (no environment, nobody logged in). Say what the user should edit in the case file,
114
+ then re-run.
115
+
116
+ Do not write anything after the `RESULT:` line.
@@ -1,137 +1,146 @@
1
- # Readiness: look at the result only after the content is there
2
-
3
- Shared by `exploring-cases`, `generating-specs` and healing. It is binding in all three.
4
-
5
- A page that has just reacted is not ready. Dialogs, composers, previews, lists and PDFs show a frame first and fill
6
- in later. Reading it too early gives a false "missing", "empty" or "wrong", and exploration then writes the false fact
7
- into the case, the spec encodes it, and healing believes it.
8
-
9
- **The rule: after every action that loads something, wait for its key requests to finish and for the content itself
10
- to be visible, before reading, asserting or recording anything.**
11
-
12
- ## What "ready" is
13
- All of these, in this order:
14
- 1. **The key requests finished** with a success status. The key requests are the data calls the action starts
15
- (the composer's content, the list query, the PDF render), not images, fonts or analytics.
16
- 2. **Nothing says it is still loading**: no spinner, progress bar, skeleton, `aria-busy="true"`, "Loading..." text.
17
- 3. **The content itself is visible**: a positive signal that only exists when the data is there (a field with its value, the
18
- first row, an image whose `naturalWidth` is above 0), never "the frame is open".
19
- 4. **Stable**: for exploration, two snapshots one second apart say the same.
20
-
21
- A check for something **missing, empty or hidden** is only valid after a positive ready signal. "The list has no X"
22
- means: the list finished loading and has rows, and none is X.
23
-
24
- ## Exploring
25
- - Know the key requests: after the action, call `browser_network_requests`, pick the data calls (XHR/fetch to the
26
- app's API, the ones that return the content). Wait until they are finished; use `browser_network_clear` before the
27
- action so the list only holds what the action started.
28
- - Then wait for the content signal (`browser_wait_for` with text, or `browser_find` that the content is there), and
29
- take the snapshot only then. A snapshot taken right after a click is a frame, not a fact.
30
- - Observations record the ready signal: `(observed after <signal>, ...)`. A fact with no signal is not recorded.
31
- - When the first read shows less than expected (empty, a missing field, a logo not there), do not record it yet and do
32
- not raise a question yet. Wait for readiness, read again, and only then decide. Mention in the report that the first
33
- read was too early, with what was missing.
34
- - Do it for every profile: loading time differs between profiles and environments.
35
-
36
- ## In the case
37
- Steps and targets state their ready signal: `Ready when: <signal>` under the step, or the target card's `Ready when`.
38
- Example: `Ready when: the composer's "To" field is visible and the logo image has loaded.` It is what `/gen-spec` turns into
39
- `ready` and `responses`.
40
-
41
- ## Generating (spec code)
42
- Use `actAndWait` and `expectNotBusy` from `@asrunjs/asrun/ready`, and `verifyDownload` from `@asrunjs/asrun/downloads`:
43
-
44
- ```ts
45
- // ready: the composer shows its recipient field and the logo image has loaded
46
- const composer = page.getByRole('dialog', {name: 'Email PDF'});
47
- await actAndWait(page, () => option.click(), {
48
- responses: [/\/api\/documents\/[^/]+\/email-preview/],
49
- ready: [composer.getByRole('textbox', {name: 'To'}), logo],
50
- });
51
- await expectImageLoaded(logo);
52
- ```
53
-
54
- - Register the response wait **before** the action (`actAndWait` does).
55
- - Every action that opens a dialog, a tab, a menu with data, a list or a preview waits for its key requests and its content
56
- signal. The request URL patterns come from `browser_network_requests` during exploration, not from guesses; if none was
57
- seen (the action was not run), say "not verified on the page" in the report.
58
- - Never `waitForTimeout`, never `networkidle`. They wait for time or for silence, not for the content. `pnpm spec:lint` fails both.
59
- - Never `if (await x.isVisible()) ...` to decide something that may not have loaded yet. Wait for a ready signal, then branch.
60
- Where an element is genuinely optional, say what proves the page finished (`// ready: ...`) before looking for it.
61
- - `locator.isVisible()` and `isHidden()` answer for this instant only. Their `timeout` option is deprecated and ignored:
62
- `isVisible({timeout: 3_000})` does not wait 3 seconds. Never write it, and never use `.catch(() => false)` to make it
63
- look like a wait. `pnpm spec:lint` fails both.
64
- - An optional confirmation (see below) is handled with `clickOptionalUntilHidden`, not with `isVisible`.
65
-
66
- ### Optional confirmations (a dialog that may or may not show up)
67
- Closing a composer, leaving an editor or deleting something can raise "Discard draft?", "Are you sure?" or nothing at all,
68
- and the dialog can show up a second or two after the click. A check made right after the click sees "no dialog" and skips
69
- it, then the next check (the composer is closed) races a dialog that appears later and fails.
70
-
71
- The click on the confirmation and the check that the action finished go into **one retry loop**:
72
-
73
- ```ts
74
- await composer.getByRole('button', {name: 'close', exact: true}).click();
75
- await clickOptionalUntilHidden(page.getByRole('button', {name: 'Discard', exact: true}), composer, {
76
- message: 'The email composer did not close',
77
- });
78
- ```
79
-
80
- - Each round: click the confirmation if it is visible now, then check that the target is hidden. A late dialog is clicked
81
- in a later round; no dialog passes in the first round.
82
- - Never decide "there is no dialog" from one look, and never wait a fixed time for it.
83
- - The same holds for any other optional modal (for example "Add to Business Tracker" after publishing): when the case says
84
- it opens, assert it with a waiting `expect(...).toBeVisible({timeout})`; when it may or may not open, use a loop that
85
- also checks the next page state, or wait for the signal that proves the action finished (URL, heading), then look.
86
- - A check for absence needs a positive ready signal in front of it (`// ready: <signal>`).
87
- - A new tab: wait for the page event first (`context.waitForEvent('page')`), then for its URL, then for its content signal.
88
- - A download is verified with `verifyDownload` (below), never with a bare `waitForEvent('download')`.
89
-
90
- ## Downloads
91
- Any download (PDF, image, CSV, ZIP), in exploration and in the spec, is verified in five parts, and the verification
92
- step fails as a whole when any part fails or times out:
93
- 1. wait for the download to start after the click;
94
- 2. wait for the download to finish;
95
- 3. check that the file exists and has content;
96
- 4. check that the file is valid for its kind (an image has the right signature; a ZIP/CSV likewise; a PDF starts with `%PDF-`,
97
- ends with `%%EOF` and **opens with at least one page**), and run the case's own validation if there is one;
98
- 5. any failure or timeout of 1 to 4 fails the verification with a message that names the stage.
99
-
100
- `verifyDownload(page, trigger, {kind, fileName?, minBytes?, validate?})` does exactly this. Name the file type in the
101
- case (`kind`), and describe in a target card what the file must contain.
102
-
103
- ### PDF content: read it like a person (rendered images and extracted text)
104
- A PDF check never stops at "it is a PDF". `verifyDownload(..., {kind: 'pdf'})` returns `file.pdf`, and
105
- `expectPdf(file.pdf!, {...})` from `@asrunjs/asrun/pdf` then:
106
- - **renders the pages to images** (scale 2, white background) and, for every region check, fails when the region is blank
107
- and compares it with the profile's baseline image (2% tolerance; the first run with `--update-snapshots` creates the
108
- baseline and a person looks at it, like any other baseline);
109
- - **extracts the text** of every page and checks `textContains` / `textNotContains` (the names of the other profiles, for
110
- example).
111
- Regions are fractions of the page from the top-left corner; `corner('top-left')` is the usual logo corner.
112
-
113
- **Which pages are checked is the case's decision, case by case:**
114
- - The case says ONLY certain pages in strong words (a requirement: "only", "exactly", "must be on the first page", "the
115
- logo is on the first and last page and nowhere else"): check just those (`pages: 'first' | 'last' | [1, 3]`), and put
116
- `// pages: <the case's words>` above the check so a reviewer can see where the limit comes from.
117
- - The case names pages softly ("should", "always", "usually", "typically", "normally", "e.g.") or names no page: check
118
- **every page** (`pages: 'all'`, the default). The named page is then only the place where the check was first seen.
119
- - A page limit that comes from the agent's own reading of the case is not allowed. When it is unclear whether a wording is
120
- a strong requirement, check all pages and put the doubt in a `[non-blocking]` TODO.
121
-
122
- **While exploring, the agent looks at the PDF too:** save the downloaded file under `.explore/` (`download.saveAs(...)`
123
- in the same `browser_run_code_unsafe` call that waits for the download), run
124
- `pnpm pdf:render <file> --out .explore/targets/<case-id>/<name> --pages all --corner top-left`, then **open the page
125
- images** and say in the card where the logo really is on each page, which pages have it and which do not, and what the
126
- text shows. The rendered pages are the card's `Reference`. If the logo is not where the case says, that is a question.
127
-
128
- While exploring, do the same by hand: click, wait for the download event, see the file finish, read its size and its first
129
- bytes. Record "(observed: <file name>, <size>, valid <kind>)".
130
-
131
- ## Healing
132
- A failure of the kind "missing", "empty", "not visible", "wrong count", "timeout waiting for X" is first a suspected
133
- readiness problem, then a suspected product problem:
134
- 1. Read the failure screenshot and video: was the page still loading (a spinner, a skeleton, an empty frame)?
135
- 2. Run the failing step by hand with the readiness protocol above and compare.
136
- 3. Only when the content is complete and still not what the case says is it a test-expectation or product matter.
137
- 4. A readiness fix goes into the spec as `actAndWait` / a ready signal, never as a longer timeout alone and never as a sleep.
1
+ # Readiness: look at the result only after the content is there
2
+
3
+ Shared by `exploring-cases`, `generating-specs` and healing. It is binding in all three.
4
+
5
+ A page that has just reacted is not ready. Dialogs, composers, previews, lists and PDFs show a frame first and fill
6
+ in later. Reading it too early gives a false "missing", "empty" or "wrong", and exploration then writes the false fact
7
+ into the case, the spec encodes it, and healing believes it.
8
+
9
+ **The rule: after every action that loads something, wait for its key requests to finish and for the content itself
10
+ to be visible, before reading, asserting or recording anything.**
11
+
12
+ ## What "ready" is
13
+ All of these, in this order:
14
+ 1. **The key requests finished** with a success status. The key requests are the data calls the action starts
15
+ (the composer's content, the list query, the PDF render), not images, fonts or analytics.
16
+ 2. **Nothing says it is still loading**: no spinner, progress bar, skeleton, `aria-busy="true"`, "Loading..." text.
17
+ 3. **The content itself is visible**: a positive signal that only exists when the data is there (a field with its value, the
18
+ first row, an image whose `naturalWidth` is above 0), never "the frame is open".
19
+ 4. **Stable**: for exploration, two snapshots one second apart say the same.
20
+
21
+ A check for something **missing, empty or hidden** is only valid after a positive ready signal. "The list has no X"
22
+ means: the list finished loading and has rows, and none is X.
23
+
24
+ ## Exploring
25
+ - Know the key requests: after the action, call `browser_network_requests`, pick the data calls (XHR/fetch to the
26
+ app's API, the ones that return the content). Wait until they are finished; use `browser_network_clear` before the
27
+ action so the list only holds what the action started.
28
+ - Then wait for the content signal (`browser_wait_for` with text, or `browser_find` that the content is there), and
29
+ take the snapshot only then. A snapshot taken right after a click is a frame, not a fact.
30
+ - Observations record the ready signal: `(observed after <signal>, ...)`. A fact with no signal is not recorded.
31
+ - When the first read shows less than expected (empty, a missing field, a logo not there), do not record it yet and do
32
+ not raise a question yet. Wait for readiness, read again, and only then decide. Mention in the report that the first
33
+ read was too early, with what was missing.
34
+ - Do it for every profile: loading time differs between profiles and environments.
35
+
36
+ ## In the case
37
+ Steps and targets state their ready signal: `Ready when: <signal>` under the step, or the target card's `Ready when`.
38
+ Example: `Ready when: the composer's "To" field is visible and the logo image has loaded.` It is what `/gen-spec` turns into
39
+ `ready` and `responses`.
40
+
41
+ **A ready signal is when to look, `## Expected` is what must hold.** Write the shortest positive signal that proves the
42
+ content arrived (one element, one image, one first row): no counts, no negatives, no "and everything else is right". The
43
+ requirement itself lives in `## Expected`, and a check may repeat the same element the signal names — one is the wait,
44
+ the other the assertion. A ready signal is never the only place where a required behaviour is written down.
45
+
46
+ A step that loads something and has no `Ready when` is read too early by whoever replays it: `/gen-spec` then waits for
47
+ the step's text alone, and a check on what only appears later fails or records a false "empty". Add one to every step
48
+ whose action opens a page, a dialog, a tab, a list or a preview.
49
+
50
+ ## Generating (spec code)
51
+ Use `actAndWait` and `expectNotBusy` from `@asrunjs/asrun/ready`, and `verifyDownload` from `@asrunjs/asrun/downloads`:
52
+
53
+ ```ts
54
+ // ready: the composer shows its recipient field and the logo image has loaded
55
+ const composer = page.getByRole('dialog', {name: 'Email PDF'});
56
+ await actAndWait(page, () => option.click(), {
57
+ responses: [/\/api\/documents\/[^/]+\/email-preview/],
58
+ ready: [composer.getByRole('textbox', {name: 'To'}), logo],
59
+ });
60
+ await expectImageLoaded(logo);
61
+ ```
62
+
63
+ - Register the response wait **before** the action (`actAndWait` does).
64
+ - Every action that opens a dialog, a tab, a menu with data, a list or a preview waits for its key requests and its content
65
+ signal. The request URL patterns come from `browser_network_requests` during exploration, not from guesses; if none was
66
+ seen (the action was not run), say "not verified on the page" in the report.
67
+ - Never `waitForTimeout`, never `networkidle`. They wait for time or for silence, not for the content. `pnpm spec:lint` fails both.
68
+ - Never `if (await x.isVisible()) ...` to decide something that may not have loaded yet. Wait for a ready signal, then branch.
69
+ Where an element is genuinely optional, say what proves the page finished (`// ready: ...`) before looking for it.
70
+ - `locator.isVisible()` and `isHidden()` answer for this instant only. Their `timeout` option is deprecated and ignored:
71
+ `isVisible({timeout: 3_000})` does not wait 3 seconds. Never write it, and never use `.catch(() => false)` to make it
72
+ look like a wait. `pnpm spec:lint` fails both.
73
+ - An optional confirmation (see below) is handled with `clickOptionalUntilHidden`, not with `isVisible`.
74
+
75
+ ### Optional confirmations (a dialog that may or may not show up)
76
+ Closing a composer, leaving an editor or deleting something can raise "Discard draft?", "Are you sure?" or nothing at all,
77
+ and the dialog can show up a second or two after the click. A check made right after the click sees "no dialog" and skips
78
+ it, then the next check (the composer is closed) races a dialog that appears later and fails.
79
+
80
+ The click on the confirmation and the check that the action finished go into **one retry loop**:
81
+
82
+ ```ts
83
+ await composer.getByRole('button', {name: 'close', exact: true}).click();
84
+ await clickOptionalUntilHidden(page.getByRole('button', {name: 'Discard', exact: true}), composer, {
85
+ message: 'The email composer did not close',
86
+ });
87
+ ```
88
+
89
+ - Each round: click the confirmation if it is visible now, then check that the target is hidden. A late dialog is clicked
90
+ in a later round; no dialog passes in the first round.
91
+ - Never decide "there is no dialog" from one look, and never wait a fixed time for it.
92
+ - The same holds for any other optional modal (for example "Add to Business Tracker" after publishing): when the case says
93
+ it opens, assert it with a waiting `expect(...).toBeVisible({timeout})`; when it may or may not open, use a loop that
94
+ also checks the next page state, or wait for the signal that proves the action finished (URL, heading), then look.
95
+ - A check for absence needs a positive ready signal in front of it (`// ready: <signal>`).
96
+ - A new tab: wait for the page event first (`context.waitForEvent('page')`), then for its URL, then for its content signal.
97
+ - A download is verified with `verifyDownload` (below), never with a bare `waitForEvent('download')`.
98
+
99
+ ## Downloads
100
+ Any download (PDF, image, CSV, ZIP), in exploration and in the spec, is verified in five parts, and the verification
101
+ step fails as a whole when any part fails or times out:
102
+ 1. wait for the download to start after the click;
103
+ 2. wait for the download to finish;
104
+ 3. check that the file exists and has content;
105
+ 4. check that the file is valid for its kind (an image has the right signature; a ZIP/CSV likewise; a PDF starts with `%PDF-`,
106
+ ends with `%%EOF` and **opens with at least one page**), and run the case's own validation if there is one;
107
+ 5. any failure or timeout of 1 to 4 fails the verification with a message that names the stage.
108
+
109
+ `verifyDownload(page, trigger, {kind, fileName?, minBytes?, validate?})` does exactly this. Name the file type in the
110
+ case (`kind`), and describe in a target card what the file must contain.
111
+
112
+ ### PDF content: read it like a person (rendered images and extracted text)
113
+ A PDF check never stops at "it is a PDF". `verifyDownload(..., {kind: 'pdf'})` returns `file.pdf`, and
114
+ `expectPdf(file.pdf!, {...})` from `@asrunjs/asrun/pdf` then:
115
+ - **renders the pages to images** (scale 2, white background) and, for every region check, fails when the region is blank
116
+ and compares it with the profile's baseline image (2% tolerance; the first run with `--update-snapshots` creates the
117
+ baseline and a person looks at it, like any other baseline);
118
+ - **extracts the text** of every page and checks `textContains` / `textNotContains` (the names of the other profiles, for
119
+ example).
120
+ Regions are fractions of the page from the top-left corner; `corner('top-left')` is the usual logo corner.
121
+
122
+ **Which pages are checked is the case's decision, case by case:**
123
+ - The case says ONLY certain pages in strong words (a requirement: "only", "exactly", "must be on the first page", "the
124
+ logo is on the first and last page and nowhere else"): check just those (`pages: 'first' | 'last' | [1, 3]`), and put
125
+ `// pages: <the case's words>` above the check so a reviewer can see where the limit comes from.
126
+ - The case names pages softly ("should", "always", "usually", "typically", "normally", "e.g.") or names no page: check
127
+ **every page** (`pages: 'all'`, the default). The named page is then only the place where the check was first seen.
128
+ - A page limit that comes from the agent's own reading of the case is not allowed. When it is unclear whether a wording is
129
+ a strong requirement, check all pages and put the doubt in a `[non-blocking]` TODO.
130
+
131
+ **While exploring, the agent looks at the PDF too:** save the downloaded file under `.explore/` (`download.saveAs(...)`
132
+ in the same `browser_run_code_unsafe` call that waits for the download), run
133
+ `pnpm pdf:render <file> --out .explore/targets/<case-id>/<name> --pages all --corner top-left`, then **open the page
134
+ images** and say in the card where the logo really is on each page, which pages have it and which do not, and what the
135
+ text shows. The rendered pages are the card's `Reference`. If the logo is not where the case says, that is a question.
136
+
137
+ While exploring, do the same by hand: click, wait for the download event, see the file finish, read its size and its first
138
+ bytes. Record "(observed: <file name>, <size>, valid <kind>)".
139
+
140
+ ## Healing
141
+ A failure of the kind "missing", "empty", "not visible", "wrong count", "timeout waiting for X" is first a suspected
142
+ readiness problem, then a suspected product problem:
143
+ 1. Read the failure screenshot and video: was the page still loading (a spinner, a skeleton, an empty frame)?
144
+ 2. Run the failing step by hand with the readiness protocol above and compare.
145
+ 3. Only when the content is complete and still not what the case says is it a test-expectation or product matter.
146
+ 4. A readiness fix goes into the spec as `actAndWait` / a ready signal, never as a longer timeout alone and never as a sleep.