@asrunjs/asrun 0.5.2 → 0.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +10 -0
- package/README.md +178 -469
- package/dist/scripts/go.mjs +188 -188
- package/dist/scripts/init.js +1 -1
- package/docs/agents/case-format.md +193 -0
- package/docs/agents/framework-dev.md +94 -6
- package/docs/ai-workflow.md +235 -0
- package/docs/cases.md +189 -0
- package/docs/commands.md +175 -0
- package/docs/install.md +174 -0
- package/docs/profiles.md +139 -0
- package/docs/troubleshooting.md +101 -0
- package/docs/web-ui.md +139 -0
- package/package.json +9 -1
- package/templates/AGENTS.md +2 -0
- package/ui/dist/assets/{dist-DSy3dVf4.js → dist-BudNAjOv.js} +1 -1
- package/ui/dist/assets/{dist-Ba-HC662.js → dist-C-eCSFja.js} +1 -1
- package/ui/dist/assets/{dist-C-Oyo1ZT.js → dist-CO4heSzh.js} +1 -1
- package/ui/dist/assets/{dist-C8MdhqgG.js → dist-CPLhRzom.js} +1 -1
- package/ui/dist/assets/{dist-Dq6sW5Ur.js → dist-DEPJZ3Me.js} +1 -1
- package/ui/dist/assets/{dist-CmpteMYY.js → dist-DZ7RiqmA.js} +1 -1
- package/ui/dist/assets/{dist-BGbe0-Aa.js → dist-LixzGniF.js} +1 -1
- package/ui/dist/assets/{dist-CG1pHn5m.js → dist-O_RVi4tP.js} +1 -1
- package/ui/dist/assets/{dist-BtMLMthJ.js → dist-jPS6HCJ5.js} +1 -1
- package/ui/dist/assets/{dist-jjxQutEm.js → dist-vmf3fnkM.js} +1 -1
- package/ui/dist/assets/{index-P_Yo0ZZn.js → index-BIGt8Hfz.js} +3 -3
- package/ui/dist/index.html +1 -1
package/README.md
CHANGED
|
@@ -1,512 +1,221 @@
|
|
|
1
1
|
# AsRun
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
`
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
- **
|
|
24
|
-
|
|
25
|
-
- **
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
`asrun` is a normal devDependency: it brings the Playwright fixtures and config, the case/spec tooling, the agent
|
|
40
|
-
prompts and the web UI. A project commits only its own test content.
|
|
3
|
+
End-to-end tests for web apps with more than one face. People write test cases in plain language; an AI agent turns
|
|
4
|
+
them into Playwright specs; every spec runs once per profile — production and staging, account A and account B — and
|
|
5
|
+
leaves a video, screenshots and a trace to review.
|
|
6
|
+
|
|
7
|
+
AsRun is a devDependency, not a service. Your profiles, cases, specs and baselines are files in your repository.
|
|
8
|
+
|
|
9
|
+
## What it does
|
|
10
|
+
|
|
11
|
+
- **Plain-language cases.** `cases/<id>.md` is the source of truth: numbered steps, expected behaviour, the exact
|
|
12
|
+
values, and the visual targets a check may look at. A person can review it without reading code.
|
|
13
|
+
- **One spec, every profile.** Each entry in `profiles.yaml` becomes a Playwright project with its own base URL,
|
|
14
|
+
saved login and screenshot baselines. Two accounts of one site give two independent runs.
|
|
15
|
+
- **An AI workflow, not a prompt dump.** `/make-case` turns raw notes into cases, `/explore-case` checks them against
|
|
16
|
+
the real pages, `/gen-spec` writes the spec, `/heal-spec` repairs it from the last report. Each command ends with
|
|
17
|
+
`RESULT: ready` or `RESULT: needs-input`, so it works unattended.
|
|
18
|
+
- **Evidence for every run.** One video per test (browser tabs joined), a trace with every action and request,
|
|
19
|
+
screenshots on failure, and a report that puts the failure first.
|
|
20
|
+
- **Safe by default.** A case is read-only unless it lists environments under `writes`; writing anywhere except
|
|
21
|
+
staging needs a person's confirmation on every run; email never leaves production; logins are done by a person, 2FA
|
|
22
|
+
included.
|
|
23
|
+
- **A local web UI.** Profiles and logins, a case editor with a form panel, AI runs with live logs, and reports. All
|
|
24
|
+
of it on `127.0.0.1`.
|
|
25
|
+
- **Fits a terminal and a pipeline.** Filter by environment, profile, tag, suite or case; one report folder per run;
|
|
26
|
+
a failing test fails the command.
|
|
27
|
+
- **No hidden state.** The framework lives in `node_modules/@asrunjs/asrun` and is never edited; a project commits
|
|
28
|
+
only its own test content.
|
|
29
|
+
|
|
30
|
+
## Requirements
|
|
31
|
+
|
|
32
|
+
- **Node.js 22+** and **pnpm**.
|
|
33
|
+
- **ffmpeg** to join the recordings of one test into a video. `pnpm go` installs it with Homebrew or winget; on Linux
|
|
34
|
+
run `sudo apt install ffmpeg`. When it is missing, the bundled `ffmpeg-static` is used; if neither works, the report
|
|
35
|
+
lists the videos separately.
|
|
36
|
+
- **Claude Code** or **OpenCode** for the AI steps. Everything else works without them.
|
|
37
|
+
|
|
38
|
+
## Install
|
|
41
39
|
|
|
42
40
|
```bash
|
|
43
|
-
pnpm dlx @asrunjs/asrun init
|
|
44
|
-
pnpm
|
|
45
|
-
pnpm exec asrun init --update # picks up the installed version
|
|
46
|
-
pnpm go # install + Chromium + doctor + web UI
|
|
41
|
+
pnpm dlx @asrunjs/asrun init --install
|
|
42
|
+
pnpm go
|
|
47
43
|
```
|
|
48
44
|
|
|
49
|
-
|
|
50
|
-
the
|
|
51
|
-
|
|
52
|
-
|
|
45
|
+
The first command adds AsRun and creates the workspace files (`profiles.yaml`, `cases/`, `tests/`, `AGENTS.md`, the
|
|
46
|
+
commands for both AI tools, `.env`, the configs); run it in the project folder, which may be empty. The second installs
|
|
47
|
+
the dependencies and Chromium, runs the checks, starts the web UI at http://127.0.0.1:4173 and opens it. After the
|
|
48
|
+
first time, `pnpm go` is enough.
|
|
53
49
|
|
|
54
|
-
|
|
50
|
+
To update:
|
|
55
51
|
|
|
56
52
|
```bash
|
|
57
|
-
pnpm
|
|
58
|
-
pnpm asrun
|
|
53
|
+
pnpm add -D @asrunjs/asrun@latest
|
|
54
|
+
pnpm asrun init --update
|
|
55
|
+
pnpm asrun doctor
|
|
59
56
|
```
|
|
60
57
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
`.env` or `node_modules/`. See "Layout" below.
|
|
64
|
-
|
|
65
|
-
**Which workspace a command runs against** comes from `.env` (`ASRUN_ROOT`, relative to that file; gitignored and
|
|
66
|
-
written by `init`), then from the nearest folder with `profiles.yaml`/`.asrun.json`. A real environment variable wins
|
|
67
|
-
over `.env`, so a CI job can point a checkout somewhere else without editing files.
|
|
58
|
+
`init --update` refreshes only the generated files; it never overwrites `AGENTS.md`, `CLAUDE.md` or your skill
|
|
59
|
+
overrides. `doctor` checks profiles, generated files, the installed version and the project rules.
|
|
68
60
|
|
|
69
|
-
|
|
61
|
+
<details>
|
|
62
|
+
<summary>Manual install or a pinned version</summary>
|
|
70
63
|
|
|
71
64
|
```bash
|
|
72
|
-
pnpm add -D @asrunjs/asrun
|
|
73
|
-
pnpm
|
|
74
|
-
pnpm
|
|
65
|
+
pnpm add -D @asrunjs/asrun # or @asrunjs/asrun@<version>
|
|
66
|
+
pnpm asrun init
|
|
67
|
+
pnpm go
|
|
75
68
|
```
|
|
76
69
|
|
|
77
|
-
|
|
78
|
-
|
|
70
|
+
On pnpm 12, run `init` before adding the dependency: it writes the `allowBuilds` entry that `esbuild` and
|
|
71
|
+
`ffmpeg-static` need, or `pnpm add` stops with `ERR_PNPM_IGNORED_BUILDS`. Details and flags:
|
|
72
|
+
[install.md](./docs/install.md).
|
|
73
|
+
|
|
74
|
+
</details>
|
|
75
|
+
|
|
76
|
+
## Your first test
|
|
77
|
+
|
|
78
|
+
1. **Point at your site.** Edit `profiles.yaml`: `id`, `name`, `env` (`staging` or `production`), `baseURL`, optional
|
|
79
|
+
`account` and `tags`. Expected values never go here; they live in the case.
|
|
80
|
+
2. **Log in once.** `pnpm signin example-staging` opens a browser. Log in yourself, then press **Done, save the
|
|
81
|
+
session** on the card in the corner. The session is saved per profile in `.auth/`.
|
|
82
|
+
3. **Write a case.** `pnpm case:new header-logo` creates `cases/header-logo.md` from the template; or start from raw
|
|
83
|
+
notes with `/make-case notes.md`. Fill in the steps, what is expected and the exact values.
|
|
84
|
+
4. **Check it against the real pages.** `/explore-case header-logo staging` opens the pages (with your saved session),
|
|
85
|
+
writes what they show into the case and turns what it cannot know into questions. Answer them in the case file, or
|
|
86
|
+
in the UI's **Answer** sheet, and run it again.
|
|
87
|
+
5. **Generate, run, read.** `/gen-spec header-logo staging` writes `tests/header-logo.spec.ts` and fills the case's
|
|
88
|
+
`## Values`. Then:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
pnpm asrun --env staging --grep header-logo
|
|
92
|
+
pnpm report
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The first run of a spec with visual checks writes its baselines: add `--update-snapshots` once, then look at every
|
|
96
|
+
new image before keeping it.
|
|
79
97
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
rule id (`F1`..`F10`) and say whether it `narrows` it (stricter) or `replaces` it (a different default, for example
|
|
83
|
-
the artifact language); `F1`..`F6` cannot be changed. A project may also refine a skill's procedure with
|
|
84
|
-
`.claude/skills/<skill>.md`. `pnpm asrun doctor` checks the table and lists the overrides, and an update never
|
|
85
|
-
rewrites a project's `AGENTS.md`, `CLAUDE.md` or overrides (`init --force` refuses them without `--reset-rules`).
|
|
98
|
+
In the web UI you never type a command: the **?** icon has the same steps with screenshots, and a case toolbar runs
|
|
99
|
+
each AI step with **Run AI**. The full procedure is in [ai-workflow.md](./docs/ai-workflow.md).
|
|
86
100
|
|
|
87
|
-
##
|
|
101
|
+
## Customize the AI for your project
|
|
88
102
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
103
|
+
The commands are thin shells; the procedure is a skill in the framework package
|
|
104
|
+
(`.claude/skills/{making-cases,exploring-cases,generating-specs}/`). A project refines a skill with one Markdown file
|
|
105
|
+
named after the skill — not after the command:
|
|
106
|
+
|
|
107
|
+
```markdown
|
|
108
|
+
<!-- .claude/skills/generating-specs.md -->
|
|
109
|
+
- Step titles in `test.step` keep the case's wording, including the number.
|
|
110
|
+
- A helper used by more than one spec goes in `tests/helpers/<name>.ts`.
|
|
111
|
+
- The staging account has no published documents: assert the empty state there instead.
|
|
92
112
|
```
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
The
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
## Every day
|
|
113
|
+
|
|
114
|
+
It is read after the framework skill and before your project rules. Write only what differs.
|
|
115
|
+
|
|
116
|
+
Rules combine in this order, and a later layer wins where they conflict:
|
|
117
|
+
|
|
118
|
+
1. `F1`–`F6` of the framework baseline (safety, in every project). They cannot be changed.
|
|
119
|
+
2. The framework baseline, `node_modules/@asrunjs/asrun/AGENTS.md`.
|
|
120
|
+
3. The framework skill.
|
|
121
|
+
4. Your override, `.claude/skills/<skill>.md`.
|
|
122
|
+
5. Your own `AGENTS.md` §2: a table of rule ids (`F7`–`F10`) with `narrows` or `replaces`, for example the artifact
|
|
123
|
+
language.
|
|
124
|
+
|
|
125
|
+
`pnpm asrun doctor` validates the table and lists the overrides it sees. An update never rewrites your rules:
|
|
126
|
+
`init --force` refuses `AGENTS.md`, `CLAUDE.md` and `.claude/skills/**` without `--reset-rules`. Never edit the
|
|
127
|
+
framework package itself: an update replaces it, and only the override files are yours. The AI tool stays configurable
|
|
128
|
+
too: `.mcp.json` (Claude Code), `opencode.jsonc` (OpenCode and its model), `.claude/settings.json` (sandbox
|
|
129
|
+
exceptions), or the model picker in the UI's top bar.
|
|
130
|
+
|
|
131
|
+
Details: [ai-workflow.md](./docs/ai-workflow.md#customizing-the-prompts).
|
|
132
|
+
|
|
133
|
+
## Every day (cookbook)
|
|
116
134
|
|
|
117
135
|
| I want to... | Command |
|
|
118
136
|
|---|---|
|
|
119
|
-
| Check the workspace
|
|
120
|
-
|
|
|
121
|
-
|
|
|
122
|
-
| Log in a profile (human, with 2FA) | `pnpm signin <profile-id>` |
|
|
137
|
+
| Check the workspace | `pnpm asrun doctor` |
|
|
138
|
+
| See which profiles are logged in | `pnpm status [--env <env>]` |
|
|
139
|
+
| Log in a profile (a person, with 2FA) | `pnpm signin <profile-id>` |
|
|
123
140
|
| Run everything in one environment | `pnpm asrun --env staging` |
|
|
124
141
|
| Run one profile | `pnpm asrun --env production --profile example-production` |
|
|
125
|
-
|
|
|
126
|
-
| Run only the profiles
|
|
142
|
+
| Leave out profiles (for example the ones not logged in) | `pnpm asrun --env production --skip-profile example-production` |
|
|
143
|
+
| Run only the profiles with a tag | `pnpm asrun --env staging --tag us` |
|
|
127
144
|
| Run only the profiles that are logged in | `pnpm asrun --env production --logged-in` |
|
|
128
|
-
|
|
|
129
|
-
|
|
|
130
|
-
| Render a PDF to images and print its text (to look at it) | `pnpm pdf:render <file.pdf> --out .explore/targets/<case-id>/<name> [--pages all] [--corner top-left]` |
|
|
131
|
-
| List screenshot baselines next to what each must show | `pnpm baselines:review [case-id] [--new]` |
|
|
132
|
-
| Overwrite the baselines of some cases | `pnpm asrun --env staging --grep <case-id> --update-baselines <case-id>` |
|
|
133
|
-
| Run one case | `pnpm asrun --env production --grep hub-landing` |
|
|
134
|
-
| Run cases of one suite | `pnpm asrun --env production --suite smoke` (several: `--suite smoke,regression`; `--priority` is the old name) |
|
|
145
|
+
| Run one case | `pnpm asrun --env production --grep header-logo` |
|
|
146
|
+
| Run a suite | `pnpm asrun --env production --suite smoke` |
|
|
135
147
|
| Leave out the cases that write data | `pnpm asrun --env production --read-only` |
|
|
136
|
-
|
|
|
148
|
+
| Look around a page as a profile | `pnpm explore example-staging /app/home` |
|
|
149
|
+
| Open the last report | `pnpm report` |
|
|
150
|
+
| Read what the last report says about a case | `pnpm heal:evidence --case <case-id> --env staging` |
|
|
151
|
+
| Check the specs | `pnpm spec:lint [case-id]` |
|
|
152
|
+
| List the baselines next to what they must show | `pnpm baselines:review [case-id] [--new]` |
|
|
153
|
+
| Update the baselines of some cases on purpose | `pnpm asrun --env staging --grep <case-id> --update-baselines <case-id>` |
|
|
154
|
+
| List test data that could not be removed | `pnpm cleanup --env staging` |
|
|
137
155
|
| Watch a run step by step | `pnpm asrun --env staging --ui` |
|
|
138
|
-
|
|
|
139
|
-
| Open the last report (videos, screenshots, traces) | `pnpm report` |
|
|
140
|
-
| Look around a page as a profile (files you download in that window are saved to `~/Downloads`; the path is printed in the terminal / job log) | `pnpm explore <profile-id> /some/path/` |
|
|
156
|
+
| Put the current step on the videos | `pnpm asrun --env production --step-text` |
|
|
141
157
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
```
|
|
148
|
-
One page for the daily work (it only listens on `127.0.0.1`):
|
|
158
|
+
The web UI: `pnpm go` (install, build, start, open), `pnpm ui` (build and serve), `pnpm ui:dev` (development with hot
|
|
159
|
+
reload). Each page covers one job: **Profiles** (login state, sign in, visit, add/duplicate/delete), **Cases** (list,
|
|
160
|
+
editor, form panel, spec view, AI runs, notes-to-cases), **Run tests** (filters, update baselines, production
|
|
161
|
+
confirmation), **Reports** (failure first, then video, screenshots, trace, full Playwright report), **Jobs** (live log
|
|
162
|
+
of every command), **Help** and **AI guide**. See [web-ui.md](./docs/web-ui.md).
|
|
149
163
|
|
|
150
|
-
|
|
151
|
-
|---|---|
|
|
152
|
-
| Profiles | Login state of every profile, one row each (same check as `pnpm status`). "Log in" starts `pnpm signin <profile-id>`; log in in the browser window that opens, then press "Done, save the session" on the card in that window's bottom-left corner (the "I've logged in, save the session" button on the job does the same, from the UI). A logged-in profile shows `Login YES`. Every row has a "..." menu: a logged-in profile gets "Verify" (same read-only session check) and "Re-login" (starts `pnpm signin` again), a saved session gets "Logout" (deletes the saved session file, after a confirmation), and every row gets "Duplicate" (adds a second entry with the same settings under a new id; the saved login is not copied) and "Delete" (removes the entry from `profiles.yaml` and deletes its saved login, after a confirmation; screenshot baselines and old reports keep the id in their names and stay on disk). "Visit" (shown for logged-in profiles) opens a logged-in browser window at the profile's `startPath`, or the site root when it is empty (`pnpm explore <profile-id> <startPath>`). While that window is open the button turns into a red "Close browser", which kills the browser process. Files downloaded in that window are saved to `~/Downloads` (an existing file is never overwritten, a ` (1)` is added) and the saved path is printed in the job log. "Add a new profile" has a form (environment, URL, account label, tags, start path) that appends an entry to `profiles.yaml`, plus a guide for what to do next (log in, update the cases, a copyable AI prompt). |
|
|
153
|
-
| Cases | List of `cases/**/*.md`, grouped by folder (collapsible), source editor with a rendered preview (both highlight what needs a human: `TODO` in any spelling such as `todo` / `TO-DO` / `to-do` (red for `🚫 TODO [blocking]`, gold otherwise), `🔍 to be confirmed by QA` / `👁️ not observed` / `TBC` / `TBD` in blue, and your `✍️ Answer:` lines in green; the emoji in front is part of the highlighted marker; hover a highlighted word for a short tooltip on what it means and how to resolve it, and click the small X after it to delete the marker, or the whole question block for a `TODO`, in one step, in the editor and in the preview (Undo brings it back; the file is only saved when you save). An `Answer:` gets its X once you typed something, and it clears what you typed), the editor and the preview scroll together, the preview block that holds the caret is shaded, and a click in the preview puts the caret on that line of the editor; `Ctrl/Cmd+S` to save, "New case" has an optional folder field and runs `pnpm case:new`. The **Spec** view is editable too (`Ctrl/Cmd+S` saves; the save is refused when the file changed on disk meanwhile): **Open in editor** opens the spec in the default application of your system for `.spec.ts` files, and **Copy path** copies its absolute path, to paste into a terminal editor such as nvim. Click the file name (or the pencil icon next to it) to rename the case or move it to another folder: the spec and the baseline images move with it and the spec is rewritten to match. **Delete** (type the id to confirm, or click the feather icon to fill it in) removes the case, its spec and its baseline images; both dialogs list exactly what they touch first, and are refused while an AI or test run is in progress. The editor toolbar has a command picker (explore / generate / heal, explore by default), an **Environment** picker (required for every command, no default), an **Attended** box (ticked by default, only for explore), a **Run AI** button (for **Heal spec** it opens a dialog first, see below) and icon buttons to copy the shell command and open the AI guide, for the one-shot AI command of the open case. The Claude / OpenCode switch (Claude by default, remembered in the browser) is in the top bar and applies to every AI command. The **Model** picker beside it chooses the model those commands run with, out of the models that CLI has on this machine (read from `opencode models`, and from Claude Code's own settings, gateway or aliases; searchable, remembered per CLI); without a choice, OpenCode's free built-in model is what runs. The **Thinking** icon beside the picker turns the agent's thinking on for every AI run and lights up while it is on (off by default). **From memo** opens a Markdown editor with its rendered preview beside it for raw test notes: **Make cases** saves them to `.tmp/memos/` and runs `/make-case` one-shot (no questions: defaults for gaps, a `TODO` block for unknown values, no page exploration, existing cases are never overwritten); **Copy /make-case** copies the memo for the full interactive dialogue. When the run ends, the **What to do next** bar shows how many cases are new; **Review** opens a sheet listing each one, with an **Explore** button that opens the case with Explore case selected: a case written from a memo has not been checked against the real pages yet. |
|
|
154
|
-
| Run tests | Form for `pnpm asrun`: environment (required, no default), profiles, suite, read-only, case, step text. The **Case** field is a searchable picker: type part of a case id and pick it from the list (only cases that have a spec are offered). The **Cases to run** list shows what the filters select, with an **Update baselines** box per case (`--update-baselines`, unticked by default; asks before it overwrites and shows the new images next to their target cards when the run ends). Asks for confirmation when write cases would run in production. |
|
|
155
|
-
| Reports | One entry per run, each with a PRODUCTION / STAGING badge (runs from before environments existed are shown as PRODUCTION). **Results** lists every case x profile with the failure reason on one line (failed first); selecting one shows a red "Why it failed" block (failed step + error message) above the screenshot, video and a folded "Technical details" (call log, stack trace). **Full Playwright report** embeds the standard HTML report (videos and traces work). The latest run opens first. Runs made before summaries existed only have the full report. |
|
|
156
|
-
| Help (the **?** icon in the top bar) | "How to start": nine short steps with screenshots, from installing AsRun to reading a report. The screenshots are in `ui/web/public/help/` (retake them when a page changes). |
|
|
157
|
-
| AI guide | How each AI step works in the Web UI (which runs the AI tool for you), and the same `/make-case`, `/explore-case`, `/gen-spec`, `/heal-spec` commands for Claude Code and OpenCode in a terminal. |
|
|
158
|
-
|
|
159
|
-
The case editor opens in **Single** with the **Panel** full width. The toolbar switches it to **Split** (source on the
|
|
160
|
-
left, one pane on the right; drag the divider to resize them, double-click it to reset), or, still in Single, to
|
|
161
|
-
**Editor** or **Doc** (the case rendered as a document). In Split the right pane is **Panel** (the default) or **Doc**;
|
|
162
|
-
its switch sits at the right end of the toolbar, because that is the pane it controls. **Panel**
|
|
163
|
-
shows the case as a form, with what needs a person first: the open questions (the text above each one is what the case
|
|
164
|
-
will say, so it is shown in full — accept it to delete the question, or write an answer and the AI folds it in; a click
|
|
165
|
-
anywhere else on a question jumps to it in the source editor and flashes the text above with the whole block, while the
|
|
166
|
-
answer field and the buttons keep their own behaviour) and the
|
|
167
|
-
`🔍 to be confirmed by QA` / `👁️ not observed` markers (click a marker to jump to its line in the source, or confirm it
|
|
168
|
-
to remove the marker and keep the sentence; the strip at the top counts them, so a collapsed list still says what
|
|
169
|
-
waits). Then the front matter (title, login, suites, writes and its note, environments), the steps (the sentence, its
|
|
170
|
-
`[writes]` / `[input]` / `[email]` tags, the `- Ready when:` lines — the signal that the content a step loads is really
|
|
171
|
-
there, which the generated test waits for before it looks — and the `- Recipe:` lines of the steps a person does: what
|
|
172
|
-
they did in the browser, recorded by `/explore-case --attended`; several lines per step are normal, and they are not
|
|
173
|
-
test data — the asserted values are under Values), the `## Expected` and `## Visual checks` bullets (drag the handle, or
|
|
174
|
-
Space and the arrow keys, to reorder them with their questions), the `## Values` table (one column per profile; an empty
|
|
175
|
-
cell inherits `default`, clearing a cell removes it from that profile), and the `## Targets` cards, read-only: one row
|
|
176
|
-
per card, with the fields it is missing and whether no check uses it. Every `[T1]` in a step, in a ready signal or next
|
|
177
|
-
to a check is a chip — orange, because it is a reference, with a white fill when the card is ready, amber when it needs
|
|
178
|
-
work and red when there is no card — and clicking it (or a Targets row) opens the card in a window: what it is, where it
|
|
179
|
-
is, how it is found, what it must not be confused with, and its reference screenshots per profile (click one for the
|
|
180
|
-
full size), with **Open in editor** to edit the text. The screenshots are local state under `.explore/` (git-ignored): on a machine
|
|
181
|
-
that never ran the exploration the card keeps the path and says the file is not there. The panel's text
|
|
182
|
-
fields (a step sentence, its ready signals and recipe lines, a bullet, an answer, the writes note) are small Markdown
|
|
183
|
-
editors with the source editor's highlighting and monospace font; they grow with their text and scroll past a height
|
|
184
|
-
limit. `## Goal`, `## Notes` and `## Source` are read-only: clicking a paragraph moves the caret of
|
|
185
|
-
the source editor to that line and flashes it three times; in **Single** a click there (and every **Open in editor**
|
|
186
|
-
button) also switches the pane back to **Editor**, so the line never opens in a pane nobody sees.
|
|
187
|
-
Every section collapses, each title carries the icon of what the section holds, and every title has a **?** icon with a
|
|
188
|
-
short help card on what it is for and how to work with it. Every edit lands in the same draft as the source editor and
|
|
189
|
-
rewrites only the bytes it changes (prose, comments and blank lines stay exactly as they are), so `Ctrl/Cmd+S` saves it
|
|
190
|
-
and **Discard** throws the panel edits away too. The model behind it is `lib/case-doc.ts`, the same parser
|
|
191
|
-
`pnpm case:check` uses.
|
|
192
|
-
|
|
193
|
-
The "Jobs" bar at the bottom (opening it shrinks the page above instead of covering it) shows every command started from the UI with live stdout/stderr and its exit
|
|
194
|
-
code; a job can be cancelled there. Only these fixed actions can run (`signin <profile>`, `visit <profile>` and `run` with
|
|
195
|
-
checked options) the three AI commands for an existing case and `/make-case` for a memo (`claude -p` / `opencode run`); there is no free shell. Set `ASRUN_UI_PORT` to change the port.
|
|
196
|
-
|
|
197
|
-
An AI job's log is read, not skimmed: a timestamp in its own style, one line per tool call (its name, what it touched and
|
|
198
|
-
how long it ran), the first lines of what that call returned with the colors a command printed, the agent's thinking
|
|
199
|
-
dimmed and italic while **Thinking** is ticked, and what the agent says rendered as Markdown. The log file kept under
|
|
200
|
-
`.tmp/ui-jobs/` holds the same text with the timestamps written out. Long output is a preview (six lines, then a count),
|
|
201
|
-
so a page snapshot or a big file cannot flood the log. When an AI job finishes, a thin **What to do next** bar appears
|
|
202
|
-
under the log; click it (or **Details**) to open its full text in a sheet from the right. The log keeps its full height,
|
|
203
|
-
so a short Jobs drawer has one scrollbar in the log, not a second one in the hint.
|
|
204
|
-
|
|
205
|
-
The **Model** picker next to the Claude / OpenCode switch chooses the model the AI commands run with. It does not
|
|
206
|
-
carry its own list: it reads the models of the CLI that is selected on this machine, so it shows the providers you
|
|
207
|
-
already signed in and the models your own configuration names.
|
|
208
|
-
- OpenCode: `opencode models`, i.e. every provider in `opencode auth list`, then passed as `--model`. Without a
|
|
209
|
-
model, a one-shot `opencode run` uses OpenCode's **built-in free model**, whose daily limit a long day of runs can
|
|
210
|
-
use up; the picker preselects the model your OpenCode TUI last used. Set `"model"` in `opencode.jsonc` to stop
|
|
211
|
-
depending on this browser's choice.
|
|
212
|
-
- Claude Code: there is no command that lists models, so the rows come from what Claude Code itself knows on this
|
|
213
|
-
machine (`model`, `modelPicker`, `ANTHROPIC_CUSTOM_MODEL_OPTION`, `ANTHROPIC_DEFAULT_*_MODEL` in its settings and
|
|
214
|
-
environment, plus the aliases `sonnet` / `opus` / `haiku` / `fable` / `opusplan`), and, for a gateway set with
|
|
215
|
-
`ANTHROPIC_BASE_URL`, from that gateway's `/v1/models` (the request Claude Code makes when its model discovery is
|
|
216
|
-
on, with your `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` / `ANTHROPIC_CUSTOM_HEADERS`). A gateway that serves
|
|
217
|
-
other models, such as Gemini, shows them too. Cloud providers (Vertex, Bedrock, Foundry) are **not** queried:
|
|
218
|
-
Claude Code does not list their models either, so only the configured ones appear there.
|
|
219
|
-
|
|
220
|
-
The choice is remembered per CLI in the browser and is shown in the command line of every AI run.
|
|
221
|
-
|
|
222
|
-
The **Thinking** icon beside the model picker (a small chip; lit while it is on) passes `--thinking` to OpenCode, so the
|
|
223
|
-
model is asked to think and the job log shows it. It is off by default: thinking costs tokens and makes a run slower.
|
|
224
|
-
Claude Code has no such flag, so the icon is disabled there; it shows thinking when its model and settings have it on.
|
|
225
|
-
Whether anything appears also depends on the model: one that does not return its reasoning shows none.
|
|
226
|
-
|
|
227
|
-
### Checking what a test actually did
|
|
228
|
-
Every run records a video and a trace per test. A new browser tab gets its own recording; the **Reports** page joins
|
|
229
|
-
them in order into one video per test (`reports/<run>/videos/`, needs ffmpeg, see Quick start). The full Playwright report
|
|
230
|
-
still lists the recordings separately. Add `--step-text`
|
|
231
|
-
to `pnpm asrun` to overlay the current step's title on the videos; without it the videos are plain.
|
|
232
|
-
In `pnpm report`, open the test: the steps are named after the steps in the case document. The trace shows each action with
|
|
233
|
-
the page before and after it, plus every network request. `pnpm asrun` also prints the file paths of the
|
|
234
|
-
videos and traces when it finishes; open a trace directly with `pnpm exec playwright show-trace <trace.zip>`.
|
|
235
|
-
Recordings contain real production pages, so do not share them.
|
|
236
|
-
|
|
237
|
-
### Logging in
|
|
238
|
-
`pnpm signin <profile-id>` opens a normal browser with a small card in its bottom-left corner. Log in yourself,
|
|
239
|
-
including 2FA, wait for the logged-in page, then press **Done, save the session** on that card (or press Enter in the
|
|
240
|
-
terminal, which does the same). The session is saved to `.auth/<profile-id>.json` and verified. Nothing about 2FA is
|
|
241
|
-
automated.
|
|
242
|
-
|
|
243
|
-
Login itself is not tested. Before every run, `pnpm asrun` checks the session of each profile of the chosen
|
|
244
|
-
environment. If a profile has no saved session, or the session expired (the check page redirects to `/login`), the
|
|
245
|
-
tests of that profile fail right away (no retries) and the run prints:
|
|
246
|
-
```
|
|
247
|
-
Not logged in (tests for these profiles will FAIL):
|
|
248
|
-
example-production expired -> pnpm signin example-production
|
|
249
|
-
Or skip them for this run: pnpm asrun --env production --skip-profile example-production
|
|
250
|
-
```
|
|
251
|
-
Either log in (`pnpm signin <profile-id>`) or leave the profile out with `--skip-profile`. Profiles that are logged
|
|
252
|
-
in are tested as usual. If a session expires in the middle of a run, the failed test is marked
|
|
253
|
-
"session expired" in the output.
|
|
254
|
-
|
|
255
|
-
Sessions are as sensitive as passwords: `.auth/` (one file per profile) is git-ignored, never share or commit it.
|
|
256
|
-
|
|
257
|
-
## From a case document to a test (AI workflow)
|
|
258
|
-
|
|
259
|
-
0. **(Optional) start from raw notes.** If you only have a memo or a list of things to check, run
|
|
260
|
-
`/make-case path/to/notes.md` (or paste the text after the command). The agent normalizes the text into steps in
|
|
261
|
-
the project's artifact language (English by default), marks steps that write data (`[writes]`) or send email
|
|
262
|
-
(`[email]`), keeps your original lines in a `## Source` table, lists the gaps with a suggested default for each,
|
|
263
|
-
and only then writes `cases/<id>.md`. It never writes specs and
|
|
264
|
-
never invents expected values (unknown ones become a default sentence with a `TODO` block under it). It never opens a
|
|
265
|
-
browser: what the pages really show is found in step 2. `pnpm case:check` validates the front matter of every case.
|
|
266
|
-
1. **Write the case.** `pnpm case:new header-logo` creates `cases/header-logo.md` from a template; `pnpm case:new documents/header-logo` puts it in the
|
|
267
|
-
group folder `cases/documents/` (created on demand, one level only). The id of a case is its path without `.md`, its spec is
|
|
268
|
-
`tests/documents/header-logo.spec.ts`, and its expected values are in its own `## Values` block.
|
|
269
|
-
Fill in steps, what is expected for every profile, and the exact values the spec asserts under Values, one entry per
|
|
270
|
-
profile id (`default` applies to all). Plain language is fine for the checks. Write it in the project's artifact
|
|
271
|
-
language (`AGENTS.md` §2; English by default); if a case is in another language, `/gen-spec` translates the case
|
|
272
|
-
file to the artifact language first (and tells you).
|
|
273
|
-
2. **(Optional) explore the case.** In OpenCode or Claude Code run `/explore-case header-logo production` (the case id is
|
|
274
|
-
required; the environment is optional: `production` or `staging`). This is the only step where a person and the
|
|
275
|
-
agent work together. The agent opens the real pages of the profiles that are logged in and only **reads** them: it
|
|
276
|
-
never creates, edits, publishes, shares or deletes anything, in staging or in production. It fixes wording in the
|
|
277
|
-
case file, writes what the pages show into it (marked `observed, 🔍 to be confirmed by QA`), and turns what it cannot
|
|
278
|
-
know into `🚫 TODO [blocking]` / `❓ TODO [non-blocking]` questions (every marker starts with its emoji). Every question is a block
|
|
279
|
-
under a sentence that is the agent's suggestion, written so that it can stand in the case as it is:
|
|
280
|
-
```
|
|
281
|
-
- Clicking "Edit Draft" opens the document editor, in the same tab or in a new tab.
|
|
282
|
-
- ❓ TODO [non-blocking]: Same tab or new tab? (👁️ not observed)
|
|
283
|
-
- ✍️ Answer:
|
|
284
|
-
```
|
|
285
|
-
It edits only the case file. **To accept the suggestion, delete the whole block** (the `TODO` line and the
|
|
286
|
-
`Answer:` line). **To give your own answer, write it after `Answer:`** (you may delete the rest of the block).
|
|
287
|
-
Then run the command again. It ends with `RESULT: ready` or `RESULT: needs-input`. Under the output of an Explore run,
|
|
288
|
-
a one-line **What to do next** bar shows what is left: the open questions (a red badge when one must be answered),
|
|
289
|
-
what the exploration marked for someone to confirm, and **Generate spec**, which opens the case with that command
|
|
290
|
-
selected. **Details** (or **Answer** when the agent waits) opens a sheet from the right with the question list, the
|
|
291
|
-
answer box and **Use the suggested defaults**, the confirmations and the generate action. A run that ends with
|
|
292
|
-
`needs-input` shows an amber "Needs your input" badge; sending answers (or "Use the suggested defaults", which
|
|
293
|
-
accepts every suggestion) continues the same agent conversation with the same permissions, folds them into the case
|
|
294
|
-
file and ends with a new `RESULT:`. Other commands with open questions show the answer box in the same sheet. A run
|
|
295
|
-
can be continued once per turn, and only while the UI server has not been restarted; after a restart, answer in the
|
|
296
|
-
case file and run the command again.
|
|
297
|
-
|
|
298
|
-
**Attended exploration (`--attended`, or the "Attended" box on the Cases page, ticked by default).** Use it for a case that creates or
|
|
299
|
-
edits data, or needs a choice (an existing document, items). Stay at the computer: the browser window opens and the
|
|
300
|
-
agent waits while you do those steps. The window shows a small card in the bottom-left corner that says what to do (drag it by its title bar or minimize it if it is in the way), with **Done** (hand
|
|
301
|
-
control back) and **Cannot do this**; the agent also continues by itself when the finish condition it named shows up
|
|
302
|
-
on the page (for example a "Saved" message), and the card turns green when control is back. It waits up to 10
|
|
303
|
-
minutes per step. Delete the `[asrun]` items you created before you press Done on the last step. What you did is
|
|
304
|
-
saved in the case under the step as **general recipe lines** (`- Recipe (phase): ...`): positions instead of names
|
|
305
|
-
("select items #1 to #3"), `[asrun]` names for created data, control labels but no selectors. Without `--attended`
|
|
306
|
-
the agent never waits: a step that needs a person becomes a `TODO [blocking]`.
|
|
307
|
-
3. **Log in the profiles the case needs** (`pnpm status`, `pnpm signin ...`). Skip for `login: none` cases.
|
|
308
|
-
4. **Generate.** In OpenCode or Claude Code run `/gen-spec header-logo staging` (the environment is required; there is
|
|
309
|
-
no default). The agent replays the case on the real pages with your saved sessions (it never writes data), writes `tests/header-logo.spec.ts`, fills the
|
|
310
|
-
case's `## Values` for the profiles it explored and checks the files (`pnpm typecheck`, `pnpm case:check`). It does not run the
|
|
311
|
-
spec and does not create screenshot baselines: running specs is a separate step (6). It does not stop to ask: what the
|
|
312
|
-
page answers is written into the case as observed (QA confirms it), and what is still unknown stays in the case as a
|
|
313
|
-
`TODO`. It ends with `RESULT: ready` or `RESULT: needs-input`. A case with a `[writes]` or `[input]` step that has no
|
|
314
|
-
`- Recipe:` line under it is not generated: it stops with `needs-input` and asks you to run step 2 with
|
|
315
|
-
`--attended`.
|
|
316
|
-
If the spec already exists, the command leaves it alone and says so. Add `--force` (`/gen-spec header-logo staging
|
|
317
|
-
--force`) to regenerate it from the case and overwrite it (the case's `## Values` and baseline images are kept; the
|
|
318
|
-
report lists the ones that may be out of date). In the web UI, **Run** on a case that has a spec
|
|
319
|
-
asks first: "Overwrite the spec?". The server refuses to start `gen-spec` on an existing spec without that choice.
|
|
320
|
-
When the run ends, **What to do next** offers **Open Run tests** with the case already selected, for step 6.
|
|
321
|
-
5. **Review what the AI produced.** Check: the spec reads like your case, and the `## Values` block of the case has the
|
|
322
|
-
right values for each profile (you normally don't edit these by hand, but you can).
|
|
323
|
-
6. **Run it:** `pnpm asrun --env <env> --grep <case-id>`. If the spec has visual checks, add `--update-snapshots`
|
|
324
|
-
the first time, then open the new images in `tests/<case>.spec.ts-snapshots/` and confirm each is the correct logo
|
|
325
|
-
for that profile. Baselines are the source of truth for pixel checks. Then run all profiles and `pnpm report`.
|
|
326
|
-
|
|
327
|
-
When the product changes and a test fails, run `/heal-spec <case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]`
|
|
328
|
-
(or **Heal spec** in the Cases toolbar, or **Heal this failure** on a failed result on the Reports page). The agent
|
|
329
|
-
decides whether the test needs updating or whether it found a real problem, and never edits an expectation just to
|
|
330
|
-
make a failure disappear. How it works:
|
|
331
|
-
- The environment is always required. Only the profiles that are logged in there are run (`pnpm asrun --logged-in`
|
|
332
|
-
leaves the others out instead of failing them).
|
|
333
|
-
- The web UI first opens a dialog: the profiles to heal (every logged-in profile ticked) and an optional text box for what
|
|
334
|
-
you want healed or any evidence and clues. The text is stored in `.tmp/hints/` (git-ignored) and passed as `--hints`.
|
|
335
|
-
Clues guide the diagnosis; they never lower the bar for a product finding.
|
|
336
|
-
- The agent reads the existing report first (`pnpm heal:evidence --case <id> --env <env>`: status, failed step, error,
|
|
337
|
-
call log and screenshots per profile, and whether the report is fresh or stale) and runs the case again only when
|
|
338
|
-
there is no report, it is stale, it does not explain the failure, or a clue has to be checked. After it edits the
|
|
339
|
-
spec it runs it once to confirm.
|
|
340
|
-
|
|
341
|
-
### Targets, readiness and downloads (why generated specs can be trusted)
|
|
342
|
-
Three rules keep an AI-written spec from passing or failing for the wrong reason. They hold in exploring, generating and
|
|
343
|
-
healing (`AGENTS.md`, `<framework>/.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`).
|
|
344
|
-
- **Targets.** A check on a logo, image, icon, file, PDF, screenshot or repeated text points to a card under `## Targets`
|
|
345
|
-
in the case (`[T1]`): what it is (and what it is not), where it is, how it is identified (never `.first()` or a bare
|
|
346
|
-
`img`), when it is ready, and a highlighted reference screenshot under `.explore/targets/`. When the agent is in doubt
|
|
347
|
-
it does not pick: it writes a `🚫 TODO [blocking]` that asks for the exact image, text, place or UI, with the
|
|
348
|
-
candidates listed. Answer it by editing the case (or, in an attended run, by outlining the target in the browser).
|
|
349
|
-
- **Readiness.** After an action that loads something, the key requests must finish and the content must be visible before
|
|
350
|
-
anything is read. Specs use `actAndWait` (`@asrunjs/asrun/ready`); "missing" and "empty" checks need a positive ready signal.
|
|
351
|
-
Healing treats a "missing / empty / timeout" failure as a readiness problem first.
|
|
352
|
-
- **Downloads.** `verifyDownload` (`@asrunjs/asrun/downloads`) checks that the download starts, finishes, the file exists and the
|
|
353
|
-
file is valid for its kind; any failure or timeout fails the whole step with a message that names the stage.
|
|
354
|
-
- **PDFs are read like a person reads them.** `expectPdf` (`@asrunjs/asrun/pdf`) renders the pages to images (a logo region must not
|
|
355
|
-
be blank and must match the profile's baseline, 2% tolerance) and extracts the text (for example, no other profile's name).
|
|
356
|
-
Every page is checked unless the case limits the pages in strong words ("only", "exactly", "must"); "should",
|
|
357
|
-
"always", "usually" or no wording mean every page.
|
|
358
|
-
`pnpm pdf:render <file.pdf> --out .explore/targets/<case-id>/<name> --corner top-left` lets you (or the exploring agent)
|
|
359
|
-
look at the rendered pages.
|
|
360
|
-
- **Checks you can run yourself:** `pnpm spec:lint` (fails on screenshots without a target, `.first()` on images, sleeps,
|
|
361
|
-
`networkidle`, raw download events, a `%PDF` check on its own), `pnpm baselines:review` (opens nothing; lists every
|
|
362
|
-
baseline with the card that says what it must show, so you can look at each new image), and `pnpm case:check`
|
|
363
|
-
(fails on an incomplete card, warns on an image check with no card).
|
|
364
|
-
|
|
365
|
-
### Updating baselines on purpose
|
|
366
|
-
If a profile really changed its logo, or a baseline is wrong: fix the spec first (the check must look at the element
|
|
367
|
-
its target card names; a wrong locator just makes a new wrong image), then
|
|
368
|
-
`pnpm asrun --env <env> --profile <id> --grep <case> --update-baselines <case-id>[,<case-id>]`, then look at the new
|
|
369
|
-
image before keeping it (`pnpm baselines:review <case-id>`).
|
|
370
|
-
- `--update-baselines` names the cases whose images are overwritten (`--update-snapshots=all`). They must be among the
|
|
371
|
-
cases the filters run. Playwright has one global switch, so they run as a second pass with their own report
|
|
372
|
-
(`<run-id>-baselines`); every other case is compared with its baselines as usual.
|
|
373
|
-
- Without `--update-baselines`, `--update-snapshots` still works as before, but it rewrites every image of the run.
|
|
374
|
-
- In the web UI, every case in **Cases to run** has an **Update baselines** box (nothing is ticked by default). A
|
|
375
|
-
confirm dialog follows, and when the run ends the job output shows each rewritten image next to the target card that
|
|
376
|
-
says what it must show.
|
|
377
|
-
- Not part of exploring, generating or healing: an agent never updates a baseline.
|
|
378
|
-
|
|
379
|
-
### Suites
|
|
380
|
-
Each case lists its suites in the front matter: `suites: [smoke, regression]` (any lowercase words). A case runs in
|
|
381
|
-
every suite it lists. The generated spec tags itself `@<suite>` for each one, which:
|
|
382
|
-
- shows as badges next to the test in the terminal and in `pnpm report`, and can be typed into the report's
|
|
383
|
-
search box (`@smoke`) to filter there;
|
|
384
|
-
- drives `pnpm asrun --suite <name>`, which can be combined with `--profile` and `--grep`. A case that is in two
|
|
385
|
-
of the chosen suites still runs once.
|
|
386
|
-
|
|
387
|
-
An unknown suite is rejected with the list of suites the cases use. The old `priority: <level>` is read as
|
|
388
|
-
`suites: [<level>]`.
|
|
389
|
-
|
|
390
|
-
### Other front matter
|
|
391
|
-
- `writes: [staging]` or `writes: [staging, production]` (omit it for a read-only case): the environments in which the
|
|
392
|
-
case may create or change data of the test accounts (and cleans it up). It then needs
|
|
393
|
-
`writes-note: <what it creates or changes>`. In an environment that is not listed, the case is **skipped with a
|
|
394
|
-
notice**, not failed. The old `writes: test-data` is refused with this hint.
|
|
395
|
-
- `envs: [production, staging]` (optional): the case runs only in these environments; in others it is skipped. Every
|
|
396
|
-
environment in `writes` must also be in `envs`.
|
|
397
|
-
|
|
398
|
-
Cases that write data get a red **WRITES** badge in the web UI and the report, and an `@writes` tag.
|
|
399
|
-
|
|
400
|
-
## What a case may and may not do
|
|
401
|
-
Tests run against production and staging. By default a case is read-only: look, read, compare. No creating,
|
|
402
|
-
editing, deleting, submitting or accepting anything.
|
|
403
|
-
|
|
404
|
-
A case may write only in the environments listed in its `writes` front matter (elsewhere it is skipped with a notice),
|
|
405
|
-
and only data of the test account (named with the prefix `[asrun]`), which it must clean up.
|
|
406
|
-
**AI agents never write data while they explore or generate**, in either environment: a person does those steps, in
|
|
407
|
-
the browser, in an attended exploration (step 2 above).
|
|
408
|
-
- **Staging:** no confirmation needed.
|
|
409
|
-
- **Production (and any environment that is not staging)**, for a case that lists it under `writes`: a person confirms every run. `pnpm asrun --env production` lists
|
|
410
|
-
the write cases and asks you to type `production`. The web UI shows a confirm dialog. Unattended runs use
|
|
411
|
-
`--allow-writes=production`; AI agents never do. `--read-only` leaves the write cases out. The permission is also
|
|
412
|
-
enforced inside the test itself, so running `playwright test` directly cannot bypass it.
|
|
413
|
-
- **Cleanup:** data is named `[asrun] ...`, registered with the `records` fixture and removed after the test, also
|
|
414
|
-
after a failure. Whatever could not be removed is shown in red in the report and listed by
|
|
415
|
-
`pnpm cleanup --env <env>`; delete it by hand, then `pnpm cleanup --env <env> --forget`.
|
|
416
|
-
- **No retries, one at a time:** write cases are never retried, and run one at a time per profile.
|
|
417
|
-
- **Email:** never sent from production. In other environments only if the profile has `email: true` in `profiles.yaml`.
|
|
418
|
-
|
|
419
|
-
## Author
|
|
420
|
-
Created by Zoro Chen.
|
|
421
|
-
|
|
422
|
-
## Layout
|
|
423
|
-
|
|
424
|
-
**In a project that uses the framework** (everything here is committed except the ignored folders):
|
|
164
|
+
## Safety
|
|
425
165
|
|
|
426
|
-
|
|
427
|
-
profiles.yaml one entry per environment: URL, account label, tags, environment (no expected values)
|
|
428
|
-
cases/ human-written cases (+ _template.md); optional group folders: cases/<group>/<name>.md
|
|
429
|
-
tests/ generated specs (same folders as cases/), and *-snapshots/ baseline images per profile
|
|
430
|
-
tsconfig.json plain compiler options: specs import the framework by package name (`@asrunjs/asrun/fixtures`)
|
|
431
|
-
playwright.config.ts re-exports node_modules/@asrunjs/asrun/dist/playwright.config.js
|
|
432
|
-
AGENTS.md the project rules: they point at the framework baseline, and §2 says what differs (narrows or replaces)
|
|
433
|
-
.claude/commands/ /make-case, /explore-case, /gen-spec, /heal-spec: thin shells that load the framework skill (Claude Code)
|
|
434
|
-
.opencode/commands/ the same commands for OpenCode (it does not read `.claude/commands`)
|
|
435
|
-
.claude/skills/ optional project overrides of a skill: <skill>.md, read by the command and the skill (not a discovered skill)
|
|
436
|
-
.mcp.json, opencode.jsonc the playwright-test MCP server, for Claude Code and OpenCode
|
|
437
|
-
.env ASRUN_ROOT (which workspace the commands use) and local settings (git-ignored)
|
|
438
|
-
node_modules/@asrunjs/asrun/ the framework (installed dependency; never edited)
|
|
439
|
-
.auth/ saved logins, one per profile (git-ignored)
|
|
440
|
-
reports/ one folder per run; reports/latest points to the newest (git-ignored)
|
|
441
|
-
```
|
|
166
|
+
Tests run against real sites, so the defaults are strict:
|
|
442
167
|
|
|
443
|
-
**
|
|
168
|
+
- **Read-only unless the case says otherwise.** A case may create, edit or delete data only in the environments it
|
|
169
|
+
lists under `writes`, and only data of the test accounts, named `[asrun] ...`, which it must clean up. Elsewhere the
|
|
170
|
+
case is skipped with a notice, not failed.
|
|
171
|
+
- **A person confirms non-staging writes on every run** by typing the environment name, or in a dialog in the UI.
|
|
172
|
+
`--read-only` leaves those cases out; `--allow-writes=<env>` is for unattended runs, and AI agents never use it.
|
|
173
|
+
- **No email from production.** Elsewhere only when the profile has `email: true`.
|
|
174
|
+
- **AI agents never write while exploring or generating.** A step that writes or needs a choice is done by a person in
|
|
175
|
+
the browser, in an attended exploration. Logins are always a person's job, 2FA included.
|
|
176
|
+
- **A failing baseline is a person's decision.** An agent never updates a baseline to make a difference disappear.
|
|
444
177
|
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
scripts/ the pnpm commands above, plus init (workspace files) and doctor (checks)
|
|
448
|
-
templates/ what `pnpm asrun init` writes into a project
|
|
449
|
-
ui/ local web UI: server/ (Hono API, job runner), web/ (React + Vite), shared/ (types)
|
|
450
|
-
docs/ static docs; theme-guide.html = the Inkstone theme (designed and created by Zoro Chen): colors, buttons, icons of the web UI (open in a browser, or via the "Inkstone theme" link in the UI's top bar)
|
|
451
|
-
examples/demo/ a committed workspace for developing the framework (public demo site; .auth/ and reports stay local)
|
|
452
|
-
.claude/ skills + slash commands for Claude Code; the skills are shared with OpenCode
|
|
453
|
-
.opencode/commands/ the same commands for OpenCode (it does not read `.claude/commands`)
|
|
454
|
-
AGENTS.md the product baseline rules (OpenCode reads it); CLAUDE.md imports it for Claude Code
|
|
455
|
-
docs/agents/ framework-dev.md: how to change asrun itself, never used by a project
|
|
456
|
-
opencode.jsonc MCP config for OpenCode; .mcp.json is the same server for Claude Code
|
|
457
|
-
```
|
|
178
|
+
The rules that enforce this are `F1`–`F10` in [AGENTS.md](./AGENTS.md); a project can tighten or replace the last
|
|
179
|
+
four, never the first six.
|
|
458
180
|
|
|
459
|
-
##
|
|
181
|
+
## Documentation
|
|
460
182
|
|
|
461
|
-
|
|
462
|
-
|
|
183
|
+
| I want to... | Read |
|
|
184
|
+
|---|---|
|
|
185
|
+
| Install, update, requirements, workspace location | [docs/install.md](./docs/install.md) |
|
|
186
|
+
| `profiles.yaml`, environments, logins, sessions | [docs/profiles.md](./docs/profiles.md) |
|
|
187
|
+
| Write or review a case | [docs/cases.md](./docs/cases.md) |
|
|
188
|
+
| The four AI commands in depth | [docs/ai-workflow.md](./docs/ai-workflow.md) |
|
|
189
|
+
| Every command and flag, reports, baselines | [docs/commands.md](./docs/commands.md) |
|
|
190
|
+
| The local web UI, page by page | [docs/web-ui.md](./docs/web-ui.md) |
|
|
191
|
+
| An error message | [docs/troubleshooting.md](./docs/troubleshooting.md) |
|
|
192
|
+
| Change AsRun itself | [docs/agents/framework-dev.md](./docs/agents/framework-dev.md) |
|
|
193
|
+
| The case document contract, for AI agents | [docs/agents/case-format.md](./docs/agents/case-format.md) |
|
|
194
|
+
|
|
195
|
+
## Maintainers
|
|
196
|
+
|
|
197
|
+
This section is for the people who maintain AsRun; the repository is private, so the instructions below assume
|
|
198
|
+
access to it. It is developed against `examples/demo`, a committed workspace whose site is a local Docker stack
|
|
199
|
+
(Conduit), so nothing runs against a public site.
|
|
463
200
|
|
|
464
201
|
```bash
|
|
465
|
-
pnpm
|
|
202
|
+
pnpm install
|
|
203
|
+
pnpm asrun init --root examples/demo
|
|
204
|
+
pnpm build # the demo's specs import the package, which resolves to dist/
|
|
205
|
+
pnpm demo:up # Docker Desktop must run
|
|
206
|
+
pnpm demo:seed
|
|
207
|
+
pnpm signin conduit-a # password in examples/demo/README.md
|
|
466
208
|
pnpm asrun --env staging --read-only
|
|
209
|
+
pnpm ui
|
|
467
210
|
```
|
|
468
211
|
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
cd examples/demo && ASRUN_ROOT=. pnpm asrun doctor
|
|
473
|
-
```
|
|
212
|
+
`pnpm demo:reset` returns the database to the state the cases expect (logins must be redone). Before a pull request:
|
|
213
|
+
`pnpm typecheck`, `pnpm build`, `pnpm ui:build`, and the checks above on the demo. A change to `templates/` or the
|
|
214
|
+
shipped docs bumps `version` in `package.json`.
|
|
474
215
|
|
|
475
|
-
|
|
476
|
-
|
|
216
|
+
Framework layout, the two agent CLIs, the UI theme and copy rules, and where documentation belongs:
|
|
217
|
+
[docs/agents/framework-dev.md](./docs/agents/framework-dev.md).
|
|
477
218
|
|
|
478
|
-
|
|
479
|
-
ASRUN_ROOT=../my-real-project pnpm asrun --env staging --read-only
|
|
480
|
-
```
|
|
219
|
+
## License
|
|
481
220
|
|
|
482
|
-
|
|
483
|
-
package is compiled to `dist/` (`pnpm build`) and the web UI to `ui/dist` (`pnpm ui`); both are shipped by `prepack`.
|
|
484
|
-
Nothing writes into `FRAMEWORK_ROOT` at run time. After changing `lib/`, run `pnpm build` before exercising the demo's
|
|
485
|
-
specs (they import the package by name, which resolves to `dist/`). After changing `templates/`, bump `version` in
|
|
486
|
-
`package.json`; `pnpm asrun doctor` in a project then shows that it is behind.
|
|
487
|
-
|
|
488
|
-
## How profiles become tests
|
|
489
|
-
Every entry in `profiles.yaml` becomes a Playwright project (name = profile id) with its own `baseURL` and saved login.
|
|
490
|
-
A spec is written once and runs once per profile; the `profile` fixture exposes that profile's `id`, `env`, `baseURL`,
|
|
491
|
-
`account`, `tags` and `email`. The expected values come from the case document, through
|
|
492
|
-
`requireValues(caseId, profile)` (its `## Values` block, `default` merged with the profile's own entry). Adding a
|
|
493
|
-
profile = add a YAML entry (by hand or with the form on the Profiles page) + `pnpm signin <id>`, then add the values
|
|
494
|
-
for it to the cases (see the guide on the Profiles page).
|
|
495
|
-
|
|
496
|
-
## Troubleshooting
|
|
497
|
-
- `Profile "x" is not logged in`: `pnpm signin <profile-id>`, or run with `--skip-profile <profile-id>`.
|
|
498
|
-
- Screenshot mismatch: open the report; the diff view shows expected / actual / diff. If the change is
|
|
499
|
-
intended, update the baseline as above.
|
|
500
|
-
- Report empty or missing: run `pnpm asrun --env <env>` first.
|
|
501
|
-
- `Choose the environment to test`: there is no default; add `--env production` or `--env staging`.
|
|
502
|
-
- An AI command says the `playwright-test` MCP server is missing: run `pnpm asrun doctor`. OpenCode reads only
|
|
503
|
-
`opencode.jsonc` and Claude Code only `.mcp.json`; `pnpm asrun init` writes both, and `opencode mcp list` /
|
|
504
|
-
`claude mcp list` shows whether the server started (`failed: ... path specified` means the `playwright` path in
|
|
505
|
-
that file does not resolve: `pnpm asrun init --force <file>`).
|
|
506
|
-
- A project rule does nothing: `AGENTS.md` §2 needs the rule id and an `Effect` of `narrows` or `replaces`; `F1`..`F6`
|
|
507
|
-
cannot be changed. `pnpm asrun doctor` names a wrong id, a missing `Effect` or a non-negotiable rule.
|
|
508
|
-
- A skill override does nothing: it is `.claude/skills/<skill>.md` (the skill name, for example `generating-specs`,
|
|
509
|
-
not the command name) and it is not discovered as a skill: the command and the skill read it. `doctor` lists the
|
|
510
|
-
overrides it sees.
|
|
511
|
-
- `pnpm asrun init --force AGENTS.md` is refused: `AGENTS.md`, `CLAUDE.md` and `.claude/skills/**` belong to the
|
|
512
|
-
project. Add `--reset-rules` to replace them with the template, and expect to lose the project's rules.
|
|
221
|
+
MIT. Created by Zoro Chen.
|