@asrunjs/asrun 0.2.0 → 0.5.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (94) hide show
  1. package/.claude/commands/explore-case.md +18 -15
  2. package/.claude/commands/gen-spec.md +16 -13
  3. package/.claude/commands/heal-spec.md +26 -23
  4. package/.claude/commands/make-case.md +11 -8
  5. package/.claude/skills/exploring-cases/SKILL.md +146 -132
  6. package/.claude/skills/exploring-cases/references/open-questions.md +116 -112
  7. package/.claude/skills/exploring-cases/references/readiness.md +146 -137
  8. package/.claude/skills/exploring-cases/references/user-handoff.md +157 -147
  9. package/.claude/skills/generating-specs/SKILL.md +219 -207
  10. package/.claude/skills/generating-specs/references/conventions.md +148 -148
  11. package/.claude/skills/making-cases/SKILL.md +72 -64
  12. package/.claude/skills/making-cases/references/normalizing.md +39 -38
  13. package/.claude/skills/making-cases/references/writes-and-email.md +37 -36
  14. package/AGENTS.md +159 -154
  15. package/CLAUDE.md +2 -1
  16. package/README.md +512 -410
  17. package/dist/lib/build-artifacts.d.ts +8 -0
  18. package/dist/lib/build-artifacts.js +10 -0
  19. package/dist/lib/case-doc.d.ts +311 -0
  20. package/dist/lib/case-doc.js +1117 -0
  21. package/dist/lib/cases.d.ts +2 -15
  22. package/dist/lib/cases.js +5 -78
  23. package/dist/lib/command.d.ts +24 -0
  24. package/dist/lib/command.js +90 -0
  25. package/dist/lib/fixtures.js +2 -2
  26. package/dist/lib/login-card.d.ts +22 -0
  27. package/dist/lib/login-card.js +183 -0
  28. package/dist/lib/node-cli.d.ts +6 -2
  29. package/dist/lib/node-cli.js +16 -6
  30. package/dist/lib/profiles.d.ts +25 -1
  31. package/dist/lib/profiles.js +54 -1
  32. package/dist/lib/run-filters.js +10 -2
  33. package/dist/lib/session.d.ts +13 -1
  34. package/dist/lib/session.js +73 -7
  35. package/dist/lib/targets.d.ts +5 -13
  36. package/dist/lib/targets.js +4 -29
  37. package/dist/lib/values.js +2 -2
  38. package/dist/scripts/build.js +2 -1
  39. package/dist/scripts/check-cases.js +74 -6
  40. package/dist/scripts/doctor.js +115 -4
  41. package/dist/scripts/explore.js +22 -1
  42. package/dist/scripts/init.d.ts +7 -1
  43. package/dist/scripts/init.js +61 -13
  44. package/dist/scripts/lint-specs.js +4 -1
  45. package/dist/scripts/login.js +30 -2
  46. package/dist/ui/server/actions.d.ts +4 -0
  47. package/dist/ui/server/actions.js +25 -12
  48. package/dist/ui/server/ai-models.d.ts +3 -0
  49. package/dist/ui/server/ai-models.js +347 -0
  50. package/dist/ui/server/ai-stream.d.ts +34 -9
  51. package/dist/ui/server/ai-stream.js +283 -77
  52. package/dist/ui/server/app.js +74 -12
  53. package/dist/ui/server/jobs.d.ts +7 -1
  54. package/dist/ui/server/jobs.js +69 -15
  55. package/dist/ui/server/profiles.d.ts +20 -1
  56. package/dist/ui/server/profiles.js +214 -38
  57. package/dist/ui/shared/ai-commands.d.ts +16 -6
  58. package/dist/ui/shared/ai-commands.js +35 -12
  59. package/dist/ui/shared/types.d.ts +114 -0
  60. package/docs/agents/framework-dev.md +79 -0
  61. package/docs/theme-guide.html +922 -692
  62. package/package.json +12 -1
  63. package/templates/.claude/commands/explore-case.md +22 -21
  64. package/templates/.claude/commands/gen-spec.md +21 -20
  65. package/templates/.claude/commands/heal-spec.md +31 -30
  66. package/templates/.claude/commands/make-case.md +13 -12
  67. package/templates/.mcp.json +8 -8
  68. package/templates/AGENTS.md +64 -45
  69. package/templates/cases/_template.md +57 -63
  70. package/templates/opencode.jsonc +16 -11
  71. package/templates/profiles.yaml +42 -29
  72. package/ui/dist/assets/{dist-BHaIOZdW.js → dist-BGbe0-Aa.js} +1 -1
  73. package/ui/dist/assets/{dist-CBV7_vDN.js → dist-Ba-HC662.js} +1 -1
  74. package/ui/dist/assets/{dist-BiE4suo9.js → dist-BtMLMthJ.js} +1 -1
  75. package/ui/dist/assets/{dist-gF2bFn-5.js → dist-C-Oyo1ZT.js} +1 -1
  76. package/ui/dist/assets/{dist-DX8knLVr.js → dist-C8MdhqgG.js} +1 -1
  77. package/ui/dist/assets/{dist-DHCY9KpV.js → dist-CG1pHn5m.js} +1 -1
  78. package/ui/dist/assets/{dist-BwYRbe5s.js → dist-CmpteMYY.js} +1 -1
  79. package/ui/dist/assets/{dist-D_I45xgJ.js → dist-DSy3dVf4.js} +1 -1
  80. package/ui/dist/assets/{dist-BqAROxzI.js → dist-Dq6sW5Ur.js} +1 -1
  81. package/ui/dist/assets/{dist-BSJMM_jp.js → dist-jjxQutEm.js} +1 -1
  82. package/ui/dist/assets/index-CTmnF0Wp.css +2 -0
  83. package/ui/dist/assets/index-P_Yo0ZZn.js +338 -0
  84. package/ui/dist/help/ai-tools.png +0 -0
  85. package/ui/dist/help/case-explore.png +0 -0
  86. package/ui/dist/help/case-generate.png +0 -0
  87. package/ui/dist/help/cases-memo.png +0 -0
  88. package/ui/dist/help/profile-form.png +0 -0
  89. package/ui/dist/help/profiles-login.png +0 -0
  90. package/ui/dist/help/reports.png +0 -0
  91. package/ui/dist/help/run-tests.png +0 -0
  92. package/ui/dist/index.html +22 -16
  93. package/ui/dist/assets/index-DmasQ6Rq.css +0 -2
  94. package/ui/dist/assets/index-I7uU93kk.js +0 -294
@@ -1,36 +1,37 @@
1
- # Steps that write data or send email
2
-
3
- ## Classify every step
4
- | Class | Looks like | Marker |
5
- |---|---|---|
6
- | read-only | open, look, read, hover, switch tab or filter, open a preview, close a modal | none |
7
- | writes | create, save, edit, publish, share, delete, upload, accept, assign a label, change a setting | `[writes]` |
8
- | input | a step that needs a person's choice or typed value (an existing document, items, a contact name), writing or not | `[input]` |
9
- | email | any step that makes the product send an email or notification (share by email, send to a recipient, invite a collaborator) | `[email]` |
10
-
11
- A step can be several of these (a step that picks items and saves is `[input] [writes]`). A step is not read-only because the notes did not say it changes something: "Share" opens a
12
- dialog (read-only) but sending the share is `[writes]` and probably `[email]`. When in doubt, mark it and ask.
13
-
14
- ## What a case with writes must look like
15
- - Front matter: `writes: [<env>, ...]`, the environments the finished spec may write in (`[staging]`; add `production`
16
- only when the notes or the user say so), and a `writes-note` that says what is created, changed and removed, in one
17
- sentence ("Creates one draft document named '[asrun] ...', publishes it, deletes it at the end").
18
- - A `## Data` section: what is created, its name, and how it is removed.
19
- - Everything created is named `[asrun] <something>` (the spec enforces this).
20
- - The case ends by deleting what it created. If the notes do not say how to delete it, that is a gap: ask, and
21
- never guess a delete path. Until it is known, write the likeliest path as a sentence in Data ("Removal: the
22
- "Delete" option of the item's menu on the list page.") with a `TODO [blocking]: How is <thing> deleted?` block
23
- under it (see `gaps.md`).
24
- - The case needs only data of the test account of the profile. It never touches data that existed before.
25
- - Do not set `envs` just because a case writes. A case that does not list an environment under `writes` is skipped
26
- there with a notice (not failed), and a human confirms a run in any environment except staging. When unsure, write
27
- `writes: [staging]`.
28
- - The agent never performs a write step while making, exploring or generating: a person does it in
29
- `/explore-case --attended`, which records a general recipe under `## Inputs`. Here, only mark the step.
30
- - Email: steps marked `[email]` are skipped by the runner in production and for profiles whose `email` is false.
31
- Write the case so that skipping them does not break the following steps, or say in Notes that it does.
32
-
33
- ## Non-linear flows
34
- Real flows often leave and come back (create a draft, stop before Publish, return to the list, check the draft,
35
- edit it again, publish). Write them as one case with the steps in the real order, including the "return to the
36
- list" steps. Do not split a flow only because it is long.
1
+ # Steps that write data or send email
2
+
3
+ ## Classify every step
4
+ | Class | Looks like | Marker |
5
+ |---|---|---|
6
+ | read-only | open, look, read, hover, switch tab or filter, open a preview, close a modal | none |
7
+ | writes | create, save, edit, publish, share, delete, upload, accept, assign a label, change a setting | `[writes]` |
8
+ | input | a step that needs a person's choice or typed value (an existing document, items, a contact name), writing or not | `[input]` |
9
+ | email | any step that makes the product send an email or notification (share by email, send to a recipient, invite a collaborator) | `[email]` |
10
+
11
+ A step can be several of these (a step that picks items and saves is `[input] [writes]`). A step is not read-only because the notes did not say it changes something: "Share" opens a
12
+ dialog (read-only) but sending the share is `[writes]` and probably `[email]`. When in doubt, mark it and ask.
13
+
14
+ ## What a case with writes must look like
15
+ - Front matter: `writes: [<env>, ...]`, the environments the finished spec may write in (`[staging]`; add `production`
16
+ only when the notes or the user say so), and a `writes-note` that says what is created, changed and removed, in one
17
+ sentence ("Creates one draft document named '[asrun] ...', publishes it, deletes it at the end"). The `writes-note`
18
+ is the only summary; there is no separate Data section, and the removal itself is a step in `## Steps`.
19
+ - Everything created is named `[asrun] <something>` (the spec enforces this).
20
+ - The case ends by deleting what it created. If the notes do not say how to delete it, that is a gap: ask, and
21
+ never guess a delete path. Until it is known, write the likeliest delete step ("Select the 'Delete' option of the
22
+ item's menu on the list page.") and put a `TODO [blocking]: How is <thing> deleted?` block under that step (see
23
+ `gaps.md`).
24
+ - The case needs only data of the test account of the profile. It never touches data that existed before.
25
+ - Do not set `envs` just because a case writes. A case that does not list an environment under `writes` is skipped
26
+ there with a notice (not failed), and a human confirms a run in any environment except staging. When unsure, write
27
+ `writes: [staging]`.
28
+ - The agent never performs a write step while making, exploring or generating: a person does it in
29
+ `/explore-case --attended`, which records a general recipe under that step as `- Recipe (phase): ...`. Here, only
30
+ mark the step.
31
+ - Email: steps marked `[email]` are skipped by the runner in production and for profiles whose `email` is false.
32
+ Write the case so that skipping them does not break the following steps, or say in Notes that it does.
33
+
34
+ ## Non-linear flows
35
+ Real flows often leave and come back (create a draft, stop before Publish, return to the list, check the draft,
36
+ edit it again, publish). Write them as one case with the steps in the real order, including the "return to the
37
+ list" steps. Do not split a flow only because it is long.
package/AGENTS.md CHANGED
@@ -1,154 +1,159 @@
1
- # AsRun — framework baseline
2
-
3
- This file is the binding baseline for every project that uses the AsRun framework, and for the framework itself.
4
- Its hard rules carry ids (`F1`..`F10`, `B1`..`B4`, `A1`..`A6`). A project may narrow one of them in its own
5
- `AGENTS.md` by naming the id; rules tagged `[non-negotiable]` may not be narrowed (`pnpm asrun doctor` checks it).
6
-
7
- Paths are relative to the **workspace root**: the project that owns `profiles.yaml`, `cases/` and `tests/`. In a
8
- project that uses asrun, the framework files are in `node_modules/@asrunjs/asrun` (an installed dependency: never edit it).
9
- `<framework>` below means that folder (`node_modules/@asrunjs/asrun` in a project, `.` in the framework repository).
10
-
11
- ## Hard rules
12
- - **F1 — Read-only by default.** [non-negotiable] Tests never create, edit, delete, submit forms that change data,
13
- send messages, or accept terms. Only a case that lists environments under `writes` in its front matter
14
- (`writes: [staging]`, `writes: [staging, production]`; with a `writes-note`) may write, only in those environments,
15
- and only data of the test accounts, which it must clean up. In any other environment the case is skipped with a
16
- notice, not failed. The `writes` environments must be ones that `envs` lets the case run in. The permission is
17
- checked at run time.
18
- - **F2 — No bypassing the write confirmation.** [non-negotiable] A case that writes in any environment except staging
19
- needs a human's confirmation on every run (typing the environment name in the CLI, or the confirm dialog in the web
20
- UI). AI agents must NEVER add `--allow-writes`, set `ASRUN_ALLOW_WRITES`, or bypass the confirmation. Use
21
- staging, or `--read-only`, instead.
22
- - **F3 — Agents never write while exploring.** [non-negotiable] A step that creates, edits, publishes, shares or
23
- deletes, and a step that needs a person's choice, is done by the person in the browser window
24
- (`/explore-case --attended`, see `<framework>/.claude/skills/exploring-cases/references/user-handoff.md`). The
25
- agent only reads. Without `--attended` an agent never waits for a person.
26
- - **F4 — No email in production.** [non-negotiable] Never send email from a test in production. In other
27
- environments only if the profile has `email: true`; new and AI-generated profile entries always get `email: false`.
28
- - **F5 — Saved logins are passwords.** [non-negotiable] Never read, print, copy or commit anything under `.auth/`.
29
- - **F6 — No 2FA automation.** [non-negotiable] Login is always done by a human via `pnpm signin <profile-id>`.
30
- - **F7 — No default environment.** `--env <env>` is required by `pnpm asrun` and by every AI command.
31
- `pnpm signin <profile-id>` and `pnpm explore <profile-id>` need no `--env`: the profile already names its
32
- environment. Sessions are saved per profile in `.auth/<profile-id>.json`.
33
- - **F8 — Login is not a test case.** `pnpm asrun` checks the session of every profile first; a profile that is not
34
- logged in (missing or expired token) fails right away with a hint to run `pnpm signin <profile-id>` or to skip it
35
- with `--skip-profile <profile-id>`. Never write specs for the login page or login flow.
36
- - **F9 — Expected values live in the case document.** They go in its `## Values` block, never in `profiles.yaml`. A
37
- profile holds connection settings only; adding or changing a profile must not touch any case value.
38
- - **F10 — Write in English.** Code, comments, docs, case files and commit messages. A Chinese version is only written
39
- when the user explicitly asks for one.
40
-
41
- ## Browser exploration rules (`B`)
42
- These apply to every agent that drives a browser to explore pages for cases or specs (`making-cases`,
43
- `exploring-cases`, `generating-specs`, the `playwright-test` MCP).
44
- - **B1 — Read-only for the agent.** Open, look, read, hover, switch tabs, open and close dialogs and menus.
45
- Everything that writes is the person's (see F3).
46
- - **B2 — Waiting for a person.** Only in an attended run. Announce the task in the browser window (a small draggable
47
- card with a Done button and click-to-copy for texts to type), name the finish condition, wait, and tell the person
48
- when control is back. Record what they did as a general recipe in the case's `## Inputs` section: positions instead
49
- of names of data that can change.
50
- - **B3 — Closing an email composer.** When closing an email composer makes a confirm dialog appear (for example
51
- "Discard draft?"), choose "Discard" and continue. Never choose "Save" or "Send"; closing the composer must not keep
52
- or send anything. The dialog can show up late: in a spec, clicking "Discard" and checking that the composer is
53
- closed happen in one retry loop (`clickOptionalUntilHidden` in `@asrunjs/asrun/ready`), never after a single
54
- `isVisible()` look.
55
- - **B4 — Record it in the case.** If the case file does not already contain this step, do not edit the flow silently.
56
- Add it to the case as a step with a `❓ TODO [non-blocking]` block for the user's confirmation, in the format of
57
- `<framework>/.claude/skills/exploring-cases/references/open-questions.md`, for example:
58
- ```
59
- 4. Close the email composer; if a confirm dialog appears, choose "Discard".
60
- - ❓ TODO [non-blocking]: Is choosing "Discard" in the close-composer confirm dialog correct for this case? (seen while exploring <profile-id>)
61
- - ✍️ Answer:
62
- ```
63
- Put the step right after the step that opens the composer (or after the last step that uses it). Mention it in the
64
- final report so the user can confirm it. If the case already has the step, just follow it and add nothing.
65
-
66
- ## Assertions and readiness (`A`, binding for exploring, generating and healing)
67
- Details: `<framework>/.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`.
68
- - **A1 — Never guess what a check looks at.** A check on a logo, image, icon, file, PDF, screenshot or repeated text
69
- names its exact target in a card in the case (`## Targets`: What, Where, Identified by, Not, Ready when, a
70
- highlighted reference screenshot, referenced as `[T<n>]`). When in doubt (several candidates, only a position or a
71
- generic locator, dynamic data in the area, no stable identity, the page differs from the case, it could not be
72
- seen), write a `🚫 TODO [blocking]` that names the exact image, text, place or UI wanted and lists the candidates.
73
- Do not pick. Look at the screenshot you saved: it must show what the card says.
74
- - **A2 — Wait for the content before you read it.** After every action that loads something, wait for its key
75
- requests to finish and for a positive content signal, then read. The same in exploration
76
- (`browser_network_requests`, `browser_wait_for`), in specs (`actAndWait` in `@asrunjs/asrun/ready`) and in healing. A
77
- "missing", "empty" or "hidden" result is only valid after a positive ready signal. Never `waitForTimeout`, never
78
- `networkidle`.
79
- - **A3 — Verify every download in five parts** with `verifyDownload` (`@asrunjs/asrun/downloads`): starts, finishes, file
80
- exists, file is valid, and any failure or timeout fails the whole step.
81
- - **A4 — A PDF is read like a person reads it.** Its pages are rendered to images (logo regions must not be blank and
82
- match the profile's baseline, 2%) and its text is extracted (`expectPdf` in `@asrunjs/asrun/pdf`; `pnpm pdf:render`
83
- while exploring). Every page is checked unless the case limits the pages in strong words ("only", "exactly",
84
- "must"); "should", "always", "usually" and no wording at all mean every page.
85
- - **A5 — No hard assertion on what nobody saw on a page.** Specs are checked with `pnpm spec:lint`; baselines with
86
- `pnpm baselines:review` (open every new image and compare it with its card).
87
- - **A6 — Observe first, delete last.** Everything that needs the data (preview, share menus, composers, downloads)
88
- is observed before the person deletes it.
89
-
90
- ## Layout
91
-
92
- **The workspace** (committed by the project; the framework never holds these):
93
- - `cases/**/*.md` human-written test cases (input of the AI workflow); `cases/<group>/<name>.md` groups an area
94
- - `tests/**/*.spec.ts` generated Playwright specs (output of the AI workflow), one per case, same folder as the case,
95
- with their `*-snapshots/` baselines next to them
96
- - `profiles.yaml` one entry per environment: URL, account label, tags, environment (connection settings only)
97
- - `.auth/` `reports/` `.tmp/` `.explore/` local state, never committed
98
-
99
- **The framework** (this folder; read-only in a project that uses it, as `node_modules/@asrunjs/asrun/`):
100
- - `lib/` fixtures (`profile`, session guard) and assertion helpers; specs import them as
101
- `@asrunjs/asrun/<module>` (for example `@asrunjs/asrun/fixtures`, `@asrunjs/asrun/values`)
102
- - `scripts/` user-facing commands, see README.md
103
- - `templates/` the files `pnpm asrun init` writes into a workspace
104
- - `ui/` local web UI (`pnpm ui`). It may only run the fixed actions in `ui/server/actions.ts`
105
- and must never serve anything under `.auth/`.
106
- - `docs/theme-guide.html` the Inkstone theme guide of the web UI: color tokens, buttons, badges, icons, surfaces
107
- - `examples/demo/` a local workspace for developing the framework itself (not test content)
108
-
109
- ## Web UI (`ui/web`)
110
- Before adding or changing any UI element, read `docs/theme-guide.html` (open the file; it is static HTML) and follow it.
111
- Its rules are binding, in particular:
112
- - Icons: Feather via the `react-feather` package only (`import {ChevronDown} from 'react-feather'`). Never draw, paste
113
- or hand-write SVGs, and never add another icon library. Size 14 in buttons, 16-20 elsewhere; decorative icons get
114
- `aria-hidden="true"`, icon-only buttons get an `aria-label`. Do not use Unicode symbols (`▲ ▼ ◀ ▶ ⤢`) as icons.
115
- - Colors: only the tokens in `ui/web/src/index.css` (the default Tailwind palette is removed). Pick the color by impact.
116
- - Buttons and badges: use `buttonClass()` and `Badge` from `ui/web/src/components/status.tsx`, never hand-style them.
117
- - Dropdowns: use `Select` from the same file, never a bare `<select>` or hand-copied classes (native arrow and height
118
- differ). A required dropdown that is empty gets `invalid` (red). See the "Dropdowns" section of the theme guide.
119
- - If a change adds or alters a token, update `ui/web/src/index.css` and `docs/theme-guide.html` together.
120
- Before reaching for a new dependency or writing a helper component, grep `package.json` and `ui/web/src` for what the
121
- project already uses.
122
-
123
- ## Workflow
124
- Three steps, each learns one thing: the intent, the facts and inputs, the implementation.
125
- 1. `making-cases` (or `/make-case <notes>`) turns raw notes into case documents. It writes `cases/*.md` only and
126
- never opens a browser. Check them with `pnpm case:check`.
127
- 2. `exploring-cases` (or `/explore-case <case-id> <env> [--attended]`) explores the real pages for one case. It
128
- edits that case file only: wording, observations, TODO blocks, and, with `--attended`, a `## Inputs` recipe of
129
- what the person did for the steps that write or need a choice.
130
- 3. `generating-specs` (or `/gen-spec <case-id> <env>`) turns the case into a spec. It replays the case and its
131
- `## Inputs` and `## Values`, writes the spec (values stay in the case document), but never runs the tests, never
132
- writes data and never creates baselines. A case with an `[input]` step that has no `## Inputs` recipe stops with
133
- `needs-input`: run `/explore-case` first. Running specs is a separate step (`pnpm asrun`, or the Run page).
134
- Do not hand-write specs unless asked. Do not edit `lib/` to make a single spec pass.
135
- Every case step is wrapped in `test.step('<n>. <case step text>', ...)` so reports and traces show the
136
- case's own steps (see the skill's `references/conventions.md`).
137
- Every AI command needs an environment (`--env`/`<env>`); there is no default and no "no environment" mode.
138
- `/heal-spec <case-id> <env>` reads the existing report first (`pnpm heal:evidence --case <id> --env <env>`), reads the
139
- user's optional clues (`--hints <file>`), runs only the profiles that are logged in (`--logged-in`) and runs the case
140
- again only when the report is missing, stale or not enough.
141
- Run `pnpm typecheck`, `pnpm spec:lint <case-id>` and `pnpm case:check` before reporting that a spec was generated. Run
142
- `pnpm asrun --env <env> --profile <id> --grep <case-id>` only when healing a spec or when asked to.
143
- Pick `--env staging` while developing. A case that does not list `production` under `writes` is skipped in
144
- production with a notice; add `--read-only` to leave all write cases out.
145
-
146
- ## Working on the framework itself
147
- - The framework repository holds no workspace content: no `profiles.yaml`, `cases/`, `tests/` or reports at the root.
148
- Develop against `examples/demo`: `pnpm asrun init --root examples/demo` (it also writes the framework's `.env`
149
- with `ASRUN_ROOT=examples/demo`), then `pnpm asrun --env staging --read-only`.
150
- - Everything the framework reads for a project goes through `ROOT` in `lib/paths.ts` (the workspace root); its own
151
- assets (templates, docs, skills, its `node_modules`) go through `FRAMEWORK_ROOT`. Never write into
152
- `FRAMEWORK_ROOT` at run time.
153
- - `templates/` is the source of the files a workspace gets. When a template changes, bump `version` in
154
- `package.json` so `pnpm asrun doctor` in a project shows that it is behind.
1
+ # AsRun — framework baseline
2
+
3
+ This file is the binding baseline for every project that uses the AsRun framework. It describes no particular
4
+ product: a project adds its own rules in its own `AGENTS.md` §2, and they win where they differ (see "Precedence").
5
+ Its hard rules carry ids (`F1`..`F10`, `B1`..`B4`, `A1`..`A6`).
6
+
7
+ Paths are relative to the **workspace root**: the project that owns `profiles.yaml`, `cases/` and `tests/`. In a
8
+ project that uses asrun, the framework files are in `node_modules/@asrunjs/asrun` (an installed dependency: never edit it).
9
+ `<framework>` below means that folder (`node_modules/@asrunjs/asrun` in a project, `.` in the framework repository).
10
+
11
+ ## Precedence
12
+
13
+ Rules are applied in this order; a later layer wins where they conflict.
14
+ 1. **F1–F6: safety.** Nothing overrides them, in any project. They are enforced in code as well
15
+ (`lib/write-gate.ts`, `lib/session.ts`, the run filters), so a rule file cannot weaken them.
16
+ 2. **This file.** The generic facts and defaults.
17
+ 3. **The framework skills** (`<framework>/.claude/skills/**`).
18
+ 4. **A project's skill override**: `.claude/skills/<skill>.md` in the workspace, if it exists. It refines that skill's
19
+ own procedure.
20
+ 5. **The project's `AGENTS.md` §2 (Project rules).** It states the project's intent and wins over layers 2–4.
21
+
22
+ A project writes §2 only for what differs from the defaults here, names the rule id, and says whether it `narrows` or
23
+ `replaces` the rule. `replaces` is refused for F1–F6; `pnpm asrun doctor` checks the table. A rule file never renames
24
+ the structural tokens under "Fixed tokens".
25
+
26
+ ## Hard rules
27
+ - **F1 — Read-only by default.** [non-negotiable] Tests never create, edit, delete, submit forms that change data,
28
+ send messages, or accept terms. Only a case that lists environments under `writes` in its front matter
29
+ (`writes: [staging]`, `writes: [staging, production]`; with a `writes-note`) may write, only in those environments,
30
+ and only data of the test accounts, which it must clean up. In any other environment the case is skipped with a
31
+ notice, not failed. The `writes` environments must be ones that `envs` lets the case run in. The permission is
32
+ checked at run time.
33
+ - **F2 — No bypassing the write confirmation.** [non-negotiable] A case that writes in any environment except staging
34
+ needs a human's confirmation on every run (typing the environment name in the CLI, or the confirm dialog in the web
35
+ UI). AI agents must NEVER add `--allow-writes`, set `ASRUN_ALLOW_WRITES`, or bypass the confirmation. Use
36
+ staging, or `--read-only`, instead.
37
+ - **F3 — Agents never write while exploring.** [non-negotiable] A step that creates, edits, publishes, shares or
38
+ deletes, and a step that needs a person's choice, is done by the person in the browser window
39
+ (`/explore-case --attended`, see `<framework>/.claude/skills/exploring-cases/references/user-handoff.md`). The
40
+ agent only reads. Without `--attended` an agent never waits for a person.
41
+ - **F4 — No email in production.** [non-negotiable] Never send email from a test in production. In other
42
+ environments only if the profile has `email: true`; new and AI-generated profile entries always get `email: false`.
43
+ - **F5 — Saved logins are passwords.** [non-negotiable] Never read, print, copy or commit anything under `.auth/`.
44
+ - **F6 — No 2FA automation.** [non-negotiable] Login is always done by a human via `pnpm signin <profile-id>`.
45
+ - **F7 — No default environment.** `--env <env>` is required by `pnpm asrun` and by every AI command.
46
+ `pnpm signin <profile-id>` and `pnpm explore <profile-id>` need no `--env`: the profile already names its
47
+ environment. Sessions are saved per profile in `.auth/<profile-id>.json`.
48
+ - **F8 — Login is not a test case.** `pnpm asrun` checks the session of every profile first; a profile that is not
49
+ logged in (missing or expired token) fails right away with a hint to run `pnpm signin <profile-id>` or to skip it
50
+ with `--skip-profile <profile-id>`. Never write specs for the login page or login flow.
51
+ - **F9 — Expected values live in the case document.** They go in its `## Values` block, never in `profiles.yaml`. A
52
+ profile holds connection settings only; adding or changing a profile must not touch any case value.
53
+ - **F10 — Artifact language.** Human-readable artifacts (case prose, report summaries, assertion messages) are
54
+ written in English by default. A project replaces this in §2 of its own `AGENTS.md` (for example Simplified Chinese
55
+ for case prose and reports). Code, comments, identifiers, commit messages and the structural tokens under
56
+ "Fixed tokens" always stay in English.
57
+
58
+ ## Browser exploration rules (`B`)
59
+ These apply to every agent that drives a browser to explore pages for cases or specs (`making-cases`,
60
+ `exploring-cases`, `generating-specs`, the `playwright-test` MCP).
61
+ - **B1 — Read-only for the agent.** Open, look, read, hover, switch tabs, open and close dialogs and menus.
62
+ Everything that writes is the person's (see F3).
63
+ - **B2 — Waiting for a person.** Only in an attended run. Announce the task in the browser window (a small draggable
64
+ card with a Done button and click-to-copy for texts to type), name the finish condition, wait, and tell the person
65
+ when control is back. Record what they did as a general recipe under the step (`- Recipe (phase): ...`): positions
66
+ instead of names of data that can change.
67
+ - **B3 — Closing an email composer.** When closing an email composer makes a confirm dialog appear (for example
68
+ "Discard draft?"), choose "Discard" and continue. Never choose "Save" or "Send"; closing the composer must not keep
69
+ or send anything. The dialog can show up late: in a spec, clicking "Discard" and checking that the composer is
70
+ closed happen in one retry loop (`clickOptionalUntilHidden` in `@asrunjs/asrun/ready`), never after a single
71
+ `isVisible()` look.
72
+ - **B4 — Record it in the case.** If the case file does not already contain this step, do not edit the flow silently.
73
+ Add it to the case as a step with a `❓ TODO [non-blocking]` block for the user's confirmation, in the format of
74
+ `<framework>/.claude/skills/exploring-cases/references/open-questions.md`, for example:
75
+ ```
76
+ 4. Close the email composer; if a confirm dialog appears, choose "Discard".
77
+ - ❓ TODO [non-blocking]: Is choosing "Discard" in the close-composer confirm dialog correct for this case? (seen while exploring <profile-id>)
78
+ - ✍️ Answer:
79
+ ```
80
+ Put the step right after the step that opens the composer (or after the last step that uses it). Mention it in the
81
+ final report so the user can confirm it. If the case already has the step, just follow it and add nothing.
82
+
83
+ ## Assertions and readiness (`A`, binding for exploring, generating and healing)
84
+ Details: `<framework>/.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`.
85
+ - **A1 — Never guess what a check looks at.** A check on a logo, image, icon, file, PDF, screenshot or repeated text
86
+ names its exact target in a card in the case (`## Targets`: What, Where, Identified by, Not, Ready when, a
87
+ highlighted reference screenshot, referenced as `[T<n>]`). When in doubt (several candidates, only a position or a
88
+ generic locator, dynamic data in the area, no stable identity, the page differs from the case, it could not be
89
+ seen), write a `🚫 TODO [blocking]` that names the exact image, text, place or UI wanted and lists the candidates.
90
+ Do not pick. Look at the screenshot you saved: it must show what the card says.
91
+ - **A2 — Wait for the content before you read it.** After every action that loads something, wait for its key
92
+ requests to finish and for a positive content signal, then read. The same in exploration
93
+ (`browser_network_requests`, `browser_wait_for`), in specs (`actAndWait` in `@asrunjs/asrun/ready`) and in healing. A
94
+ "missing", "empty" or "hidden" result is only valid after a positive ready signal. Never `waitForTimeout`, never
95
+ `networkidle`.
96
+ - **A3 — Verify every download in five parts** with `verifyDownload` (`@asrunjs/asrun/downloads`): starts, finishes, file
97
+ exists, file is valid, and any failure or timeout fails the whole step.
98
+ - **A4 — A PDF is read like a person reads it.** Its pages are rendered to images (logo regions must not be blank and
99
+ match the profile's baseline, 2%) and its text is extracted (`expectPdf` in `@asrunjs/asrun/pdf`; `pnpm pdf:render`
100
+ while exploring). Every page is checked unless the case limits the pages in strong words ("only", "exactly",
101
+ "must"); "should", "always", "usually" and no wording at all mean every page.
102
+ - **A5 — No hard assertion on what nobody saw on a page.** Specs are checked with `pnpm spec:lint`; baselines with
103
+ `pnpm baselines:review` (open every new image and compare it with its card).
104
+ - **A6 — Observe first, delete last.** Everything that needs the data (preview, share menus, composers, downloads) is
105
+ observed before the person deletes it.
106
+
107
+ ## Fixed tokens
108
+ `lib/case-doc.ts`, the scripts and the web panel parse these literals. A project language never translates them, and
109
+ a project rule never renames them.
110
+ - Section headings: `## Goal`, `## Steps`, `## Expected`, `## Values`, `## Visual checks`, `## Targets`, `## Notes`,
111
+ `## Source` (only when the case was made from notes).
112
+ - A step's recipe lines: `- Recipe (phase): ...` under the step it belongs to.
113
+ - `### T<n>` target cards and the `[T<n>]` references to them.
114
+ - Question blocks: `🚫 TODO [blocking]`, `❓ TODO [non-blocking]`, `✍️ Answer:` (with its emoji), and the markers
115
+ `🔍 to be confirmed by QA` / `👁️ not observed`.
116
+ - Step tags `[input]`, `[writes]`, `[email]`, and the last line `RESULT: ready` / `RESULT: needs-input`.
117
+ - Front-matter keys: `id`, `title`, `login`, `suites`, `writes`, `writes-note`, `envs`.
118
+
119
+ ## Layout
120
+
121
+ **The workspace** (committed by the project; the framework never holds these):
122
+ - `cases/**/*.md` human-written test cases (input of the AI workflow); `cases/<group>/<name>.md` groups an area
123
+ - `tests/**/*.spec.ts` generated Playwright specs (output of the AI workflow), one per case, same folder as the case,
124
+ with their `*-snapshots/` baselines next to them
125
+ - `profiles.yaml` one entry per environment: URL, account label, tags, environment (connection settings only)
126
+ - `AGENTS.md` the project's own rules: they point at this baseline, and §2 holds what differs
127
+ - `.auth/` `reports/` `.tmp/` `.explore/` local state, never committed
128
+
129
+ The framework folder is read-only in a project: never edit it. Its own layout and how to change it are in
130
+ `docs/agents/framework-dev.md`.
131
+
132
+ ## Workflow
133
+ Three steps, each learns one thing: the intent, the facts and inputs, the implementation.
134
+ 1. `making-cases` (or `/make-case <notes>`) turns raw notes into case documents. It writes `cases/*.md` only and
135
+ never opens a browser. Check them with `pnpm case:check`.
136
+ 2. `exploring-cases` (or `/explore-case <case-id> <env> [--attended]`) explores the real pages for one case. It
137
+ edits that case file only: wording, observations, TODO blocks, and, with `--attended`, the `- Recipe:` lines of
138
+ the steps that write or need a choice.
139
+ 3. `generating-specs` (or `/gen-spec <case-id> <env>`) turns the case into a spec. It replays the case and its
140
+ step recipes and `## Values`, writes the spec (values stay in the case document), but never runs the tests, never
141
+ writes data and never creates baselines. A tagged step with no `- Recipe:` line (an old `## Inputs` line counts
142
+ until it is moved) stops with `needs-input`: run `/explore-case` first. Running specs is a separate step
143
+ (`pnpm asrun`, or the Run page).
144
+ Do not hand-write specs unless asked. Do not edit `lib/` to make a single spec pass.
145
+ Every case step is wrapped in `test.step('<n>. <case step text>', ...)` so reports and traces show the
146
+ case's own steps (see the skill's `references/conventions.md`).
147
+ Every AI command needs an environment (`--env`/`<env>`); there is no default and no "no environment" mode.
148
+ `/heal-spec <case-id> <env>` reads the existing report first (`pnpm heal:evidence --case <id> --env <env>`), reads the
149
+ user's optional clues (`--hints <file>`), runs only the profiles that are logged in (`--logged-in`) and runs the case
150
+ again only when the report is missing, stale or not enough.
151
+ Run `pnpm typecheck`, `pnpm spec:lint <case-id>` and `pnpm case:check` before reporting that a spec was generated. Run
152
+ `pnpm asrun --env <env> --profile <id> --grep <case-id>` only when healing a spec or when asked to.
153
+ Pick `--env staging` while developing. A case that does not list `production` under `writes` is skipped in
154
+ production with a notice; add `--read-only` to leave all write cases out.
155
+
156
+ ## Working on the framework itself
157
+ This file is the product baseline. How to change asrun (repository layout, templates and versioning, the web UI
158
+ theme, developing against `examples/demo`) is in `docs/agents/framework-dev.md`; read it only when working in the
159
+ framework repository.
package/CLAUDE.md CHANGED
@@ -1 +1,2 @@
1
- @AGENTS.md
1
+ @AGENTS.md
2
+ @docs/agents/framework-dev.md