@asrunjs/asrun 0.5.2 → 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,512 +1,221 @@
1
1
  # AsRun
2
2
 
3
- Automated tests against production and staging for multiple profiles. Humans write test cases in plain language; an AI
4
- agent turns them into Playwright specs; the specs run locally, once per profile, and produce videos, screenshots and
5
- traces for review.
6
-
7
- Profiles and environments: see `profiles.yaml`. A profile is one environment of the system under test (a URL, an
8
- account label, tags and an environment name), and each one becomes one Playwright project.
9
-
10
- ## Profiles and environments
11
- Every entry in `profiles.yaml` is one environment: `id`, `name`, `env`, `baseURL`, optional `account`, `tags`,
12
- `startPath`, `sessionCheckPath` and `sessionCheck`, and `email`. One profile = one Playwright project named after its
13
- id (for example `example-staging`).
14
- - **There is no default environment for a run.** `pnpm asrun` and every AI command need `--env <env>`, which selects
15
- the profiles whose `env` matches (the Run tests page needs one radio button picked). `pnpm signin <profile-id>` and
16
- `pnpm explore <profile-id>` need no `--env`: the profile already names its environment.
17
- - **Separate logins.** Sessions are saved per profile in `.auth/<profile-id>.json`; log in with
18
- `pnpm signin <profile-id>` (a human, including 2FA).
19
- - **Separate baselines and reports.** Screenshot baselines are stored per project (`...-example-staging-darwin.png`).
20
- A run's folder, report title and summary carry the environment (`reports/2026-10-28T07-30-22-staging`); the web UI
21
- shows PRODUCTION in red and STAGING in blue everywhere, and prints `ENV: PRODUCTION` / `ENV: STAGING` at the start of a CLI run.
22
- - **Tags.** `tags: [us, core]` groups profiles; `pnpm asrun --env staging --tag us` runs only the matching ones.
23
- - **Email.** `email: true` says a real mailbox is connected and tests may send email. It defaults to `false`, is set to
24
- `false` for new profiles, and is rejected for `production`.
25
- - **Visit start path.** `startPath` (`/app/home`) is the page the Profiles page's "Visit" opens; empty opens the site
26
- root. It must be a path on the site (a single leading `/`, no spaces), so it can never leave the site.
27
- `pnpm explore <profile-id>` without a path still opens the root. `sessionCheckPath` (default `/account/`) is the
28
- page a saved session is checked against, and `sessionCheck` says how that page shows a missing login: the
29
- address a signed-out visitor is sent to (default `/login`), a selector that only exists while signed in, a
30
- selector that proves the visitor is signed out, and how long to wait for one of them. Sites differ: a
31
- single-page app redirects after it has started, some answer `401`, some show the login form at the same
32
- address. Without a `loggedInSelector`, "not sent to the login page" counts as signed in, which can mislead on
33
- a slow single-page app.
34
- - **Expected values are not here.** They live in each case document (`## Values`); `profiles.yaml` holds connection
35
- settings only, so adding a profile never touches a case and two profiles can never fight over one case's values.
36
-
37
- ## Installing asrun in a project
38
-
39
- `asrun` is a normal devDependency: it brings the Playwright fixtures and config, the case/spec tooling, the agent
40
- prompts and the web UI. A project commits only its own test content.
3
+ End-to-end tests for web apps with more than one face. People write test cases in plain language; an AI agent turns
4
+ them into Playwright specs; every spec runs once per profile — production and staging, account A and account B — and
5
+ leaves a video, screenshots and a trace to review.
6
+
7
+ AsRun is a devDependency, not a service. Your profiles, cases, specs and baselines are files in your repository.
8
+
9
+ ## What it does
10
+
11
+ - **Plain-language cases.** `cases/<id>.md` is the source of truth: numbered steps, expected behaviour, the exact
12
+ values, and the visual targets a check may look at. A person can review it without reading code.
13
+ - **One spec, every profile.** Each entry in `profiles.yaml` becomes a Playwright project with its own base URL,
14
+ saved login and screenshot baselines. Two accounts of one site give two independent runs.
15
+ - **An AI workflow, not a prompt dump.** `/make-case` turns raw notes into cases, `/explore-case` checks them against
16
+ the real pages, `/gen-spec` writes the spec, `/heal-spec` repairs it from the last report. Each command ends with
17
+ `RESULT: ready` or `RESULT: needs-input`, so it works unattended.
18
+ - **Evidence for every run.** One video per test (browser tabs joined), a trace with every action and request,
19
+ screenshots on failure, and a report that puts the failure first.
20
+ - **Safe by default.** A case is read-only unless it lists environments under `writes`; writing anywhere except
21
+ staging needs a person's confirmation on every run; email never leaves production; logins are done by a person, 2FA
22
+ included.
23
+ - **A local web UI.** Profiles and logins, a case editor with a form panel, AI runs with live logs, and reports. All
24
+ of it on `127.0.0.1`.
25
+ - **Fits a terminal and a pipeline.** Filter by environment, profile, tag, suite or case; one report folder per run;
26
+ a failing test fails the command.
27
+ - **No hidden state.** The framework lives in `node_modules/@asrunjs/asrun` and is never edited; a project commits
28
+ only its own test content.
29
+
30
+ ## Requirements
31
+
32
+ - **Node.js 22+** and **pnpm**.
33
+ - **ffmpeg** to join the recordings of one test into a video. `pnpm go` installs it with Homebrew or winget; on Linux
34
+ run `sudo apt install ffmpeg`. When it is missing, the bundled `ffmpeg-static` is used; if neither works, the report
35
+ lists the videos separately.
36
+ - **Claude Code** or **OpenCode** for the AI steps. Everything else works without them.
37
+
38
+ ## Install
41
39
 
42
40
  ```bash
43
- pnpm dlx @asrunjs/asrun init # scaffolding only: writes .env, configs, AGENTS.md, commands, the build-script approval
44
- pnpm add -D @asrunjs/asrun # or: pnpm add -D github:wizcas/asrun#v0.1.0
45
- pnpm exec asrun init --update # picks up the installed version
46
- pnpm go # install + Chromium + doctor + web UI
41
+ pnpm dlx @asrunjs/asrun init --install
42
+ pnpm go
47
43
  ```
48
44
 
49
- Running `init` before `add` matters on pnpm 12: it writes the `allowBuilds` entry that pnpm needs before it will run
50
- the install scripts of `esbuild`/`ffmpeg-static` (without it, `pnpm add` exits with `ERR_PNPM_IGNORED_BUILDS`).
51
- `pnpm dlx @asrunjs/asrun init --install` does both in one step (the spec is optional:
52
- `--install github:wizcas/asrun#v0.1.0`).
45
+ The first command adds AsRun and creates the workspace files (`profiles.yaml`, `cases/`, `tests/`, `AGENTS.md`, the
46
+ commands for both AI tools, `.env`, the configs); run it in the project folder, which may be empty. The second installs
47
+ the dependencies and Chromium, runs the checks, starts the web UI at http://127.0.0.1:4173 and opens it. After the
48
+ first time, `pnpm go` is enough.
53
49
 
54
- `pnpm asrun init` is safe to run again: it never overwrites a file you own.
50
+ To update:
55
51
 
56
52
  ```bash
57
- pnpm asrun init --update # refreshes only the generated files (playwright.config.ts, seed.spec.ts, commands)
58
- pnpm asrun doctor # checks profiles, the tsconfig alias, generated files, the installed version, skill overrides
53
+ pnpm add -D @asrunjs/asrun@latest
54
+ pnpm asrun init --update
55
+ pnpm asrun doctor
59
56
  ```
60
57
 
61
- The project commits `profiles.yaml`, `cases/`, `tests/` (with their `*-snapshots/`), `AGENTS.md`, `.claude/`,
62
- `.mcp.json`, `opencode.jsonc` and the generated files. It never commits `.auth/`, `reports/`, `.tmp/`, `.explore/`,
63
- `.env` or `node_modules/`. See "Layout" below.
64
-
65
- **Which workspace a command runs against** comes from `.env` (`ASRUN_ROOT`, relative to that file; gitignored and
66
- written by `init`), then from the nearest folder with `profiles.yaml`/`.asrun.json`. A real environment variable wins
67
- over `.env`, so a CI job can point a checkout somewhere else without editing files.
58
+ `init --update` refreshes only the generated files; it never overwrites `AGENTS.md`, `CLAUDE.md` or your skill
59
+ overrides. `doctor` checks profiles, generated files, the installed version and the project rules.
68
60
 
69
- Update:
61
+ <details>
62
+ <summary>Manual install or a pinned version</summary>
70
63
 
71
64
  ```bash
72
- pnpm add -D @asrunjs/asrun@latest # or: pnpm add -D github:wizcas/asrun#v0.2.0
73
- pnpm exec asrun init --update
74
- pnpm asrun doctor
65
+ pnpm add -D @asrunjs/asrun # or @asrunjs/asrun@<version>
66
+ pnpm asrun init
67
+ pnpm go
75
68
  ```
76
69
 
77
- There is no folder to move: the framework lives in `node_modules/@asrunjs/asrun`, which the package manager owns. To change
78
- the framework itself, `pnpm patch asrun` (a committed patch) or point the dependency at your fork's tag.
70
+ On pnpm 12, run `init` before adding the dependency: it writes the `allowBuilds` entry that `esbuild` and
71
+ `ffmpeg-static` need, or `pnpm add` stops with `ERR_PNPM_IGNORED_BUILDS`. Details and flags:
72
+ [install.md](./docs/install.md).
73
+
74
+ </details>
75
+
76
+ ## Your first test
77
+
78
+ 1. **Point at your site.** Edit `profiles.yaml`: `id`, `name`, `env` (`staging` or `production`), `baseURL`, optional
79
+ `account` and `tags`. Expected values never go here; they live in the case.
80
+ 2. **Log in once.** `pnpm signin example-staging` opens a browser. Log in yourself, then press **Done, save the
81
+ session** on the card in the corner. The session is saved per profile in `.auth/`.
82
+ 3. **Write a case.** `pnpm case:new header-logo` creates `cases/header-logo.md` from the template; or start from raw
83
+ notes with `/make-case notes.md`. Fill in the steps, what is expected and the exact values.
84
+ 4. **Check it against the real pages.** `/explore-case header-logo staging` opens the pages (with your saved session),
85
+ writes what they show into the case and turns what it cannot know into questions. Answer them in the case file, or
86
+ in the UI's **Answer** sheet, and run it again.
87
+ 5. **Generate, run, read.** `/gen-spec header-logo staging` writes `tests/header-logo.spec.ts` and fills the case's
88
+ `## Values`. Then:
89
+
90
+ ```bash
91
+ pnpm asrun --env staging --grep header-logo
92
+ pnpm report
93
+ ```
94
+
95
+ The first run of a spec with visual checks writes its baselines: add `--update-snapshots` once, then look at every
96
+ new image before keeping it.
79
97
 
80
- `templates/` in the package holds what `init` writes. `AGENTS.md` and the skills stay in the package as the single
81
- source and carry no product-specific rules. A project states what differs in §2 of its own `AGENTS.md`: name the
82
- rule id (`F1`..`F10`) and say whether it `narrows` it (stricter) or `replaces` it (a different default, for example
83
- the artifact language); `F1`..`F6` cannot be changed. A project may also refine a skill's procedure with
84
- `.claude/skills/<skill>.md`. `pnpm asrun doctor` checks the table and lists the overrides, and an update never
85
- rewrites a project's `AGENTS.md`, `CLAUDE.md` or overrides (`init --force` refuses them without `--reset-rules`).
98
+ In the web UI you never type a command: the **?** icon has the same steps with screenshots, and a case toolbar runs
99
+ each AI step with **Run AI**. The full procedure is in [ai-workflow.md](./docs/ai-workflow.md).
86
100
 
87
- ## Quick start in a project
101
+ ## Customize the AI for your project
88
102
 
89
- ```bash
90
- node framework/scripts/go.mjs # the very first time (no dependencies yet)
91
- pnpm go # every time after that
103
+ The commands are thin shells; the procedure is a skill in the framework package
104
+ (`.claude/skills/{making-cases,exploring-cases,generating-specs}/`). A project refines a skill with one Markdown file
105
+ named after the skill — not after the command:
106
+
107
+ ```markdown
108
+ <!-- .claude/skills/generating-specs.md -->
109
+ - Step titles in `test.step` keep the case's wording, including the number.
110
+ - A helper used by more than one spec goes in `tests/helpers/<name>.ts`.
111
+ - The staging account has no published documents: assert the empty state there instead.
92
112
  ```
93
- That is all. It checks Node (22+), lists the framework in `pnpm-workspace.yaml`, installs the dependencies and the
94
- browser (`pnpm install` + Chromium), writes `.env` and the workspace files, runs `doctor`, starts the web UI and opens
95
- http://127.0.0.1:4173. Run it again any day: the install step then takes seconds, and if the UI is already running it
96
- just opens it. The **?** icon in the top bar of the UI opens the step-by-step "How to start" page (install AsRun in a
97
- project, choose the AI tool, add a profile, log in, make cases, explore, generate the spec, run the tests, read the
98
- report). `pnpm go --no-open` does not open the browser.
99
-
100
- The tab title shows where you are (`AsRun | Cases | documents/landing-page`, `AsRun | Job Log`). When it is your
101
- turn (a login to finish, a step that an attended exploration hands to you in the browser window, or answers that an
102
- AI run needs), the title gets a count in front (`(1) AsRun | ...`), the job shows a "Your turn" badge, and the bell
103
- icon in the top bar, once switched on, shows a browser notification: it comes when the tab goes to the background (or
104
- at once, if it already was) and stays until you click or close it. A test run or an AI run that ends while the tab is
105
- in the background is announced the same way. Clicking a notification opens that job.
106
-
107
- You need Node 22+ and pnpm. The AI steps (make cases, explore, generate) also need Claude Code or OpenCode; `pnpm go`
108
- warns if neither is found, everything else works without them.
109
- `ffmpeg` is also needed: it joins the videos of one test (a new browser tab has its own recording) into a single video
110
- for the report. `pnpm go` tries to install it (Homebrew on macOS, winget on Windows; on Linux run
111
- `sudo apt install ffmpeg`). If it is missing, the optional `ffmpeg-static` package is used instead; if neither works,
112
- the report shows the videos separately.
113
- Then check `profiles.yaml`: the profile ids, names, environments and addresses.
114
-
115
- ## Every day
113
+
114
+ It is read after the framework skill and before your project rules. Write only what differs.
115
+
116
+ Rules combine in this order, and a later layer wins where they conflict:
117
+
118
+ 1. `F1`–`F6` of the framework baseline (safety, in every project). They cannot be changed.
119
+ 2. The framework baseline, `node_modules/@asrunjs/asrun/AGENTS.md`.
120
+ 3. The framework skill.
121
+ 4. Your override, `.claude/skills/<skill>.md`.
122
+ 5. Your own `AGENTS.md` §2: a table of rule ids (`F7`–`F10`) with `narrows` or `replaces`, for example the artifact
123
+ language.
124
+
125
+ `pnpm asrun doctor` validates the table and lists the overrides it sees. An update never rewrites your rules:
126
+ `init --force` refuses `AGENTS.md`, `CLAUDE.md` and `.claude/skills/**` without `--reset-rules`. Never edit the
127
+ framework package itself: an update replaces it, and only the override files are yours. The AI tool stays configurable
128
+ too: `.mcp.json` (Claude Code), `opencode.jsonc` (OpenCode and its model), `.claude/settings.json` (sandbox
129
+ exceptions), or the model picker in the UI's top bar.
130
+
131
+ Details: [ai-workflow.md](./docs/ai-workflow.md#customizing-the-prompts).
132
+
133
+ ## Every day (cookbook)
116
134
 
117
135
  | I want to... | Command |
118
136
  |---|---|
119
- | Check the workspace (profiles, tsconfig alias, generated files, skill overrides) | `pnpm asrun doctor` |
120
- | Create or refresh the generated workspace files | `pnpm asrun init`, `pnpm asrun init --update` |
121
- | See which profiles are logged in | `pnpm status` (all environments) or `pnpm status --env staging` |
122
- | Log in a profile (human, with 2FA) | `pnpm signin <profile-id>` |
137
+ | Check the workspace | `pnpm asrun doctor` |
138
+ | See which profiles are logged in | `pnpm status [--env <env>]` |
139
+ | Log in a profile (a person, with 2FA) | `pnpm signin <profile-id>` |
123
140
  | Run everything in one environment | `pnpm asrun --env staging` |
124
141
  | Run one profile | `pnpm asrun --env production --profile example-production` |
125
- | Run all profiles except some (e.g. not logged in) | `pnpm asrun --env production --skip-profile example-production,example-staging` |
126
- | Run only the profiles that carry a tag | `pnpm asrun --env staging --tag us` |
142
+ | Leave out profiles (for example the ones not logged in) | `pnpm asrun --env production --skip-profile example-production` |
143
+ | Run only the profiles with a tag | `pnpm asrun --env staging --tag us` |
127
144
  | Run only the profiles that are logged in | `pnpm asrun --env production --logged-in` |
128
- | Read what the last report says about a case | `pnpm heal:evidence --case <case-id> --env staging` |
129
- | Check the specs for guessed targets, sleeps and weak download checks | `pnpm spec:lint [case-id]` |
130
- | Render a PDF to images and print its text (to look at it) | `pnpm pdf:render <file.pdf> --out .explore/targets/<case-id>/<name> [--pages all] [--corner top-left]` |
131
- | List screenshot baselines next to what each must show | `pnpm baselines:review [case-id] [--new]` |
132
- | Overwrite the baselines of some cases | `pnpm asrun --env staging --grep <case-id> --update-baselines <case-id>` |
133
- | Run one case | `pnpm asrun --env production --grep hub-landing` |
134
- | Run cases of one suite | `pnpm asrun --env production --suite smoke` (several: `--suite smoke,regression`; `--priority` is the old name) |
145
+ | Run one case | `pnpm asrun --env production --grep header-logo` |
146
+ | Run a suite | `pnpm asrun --env production --suite smoke` |
135
147
  | Leave out the cases that write data | `pnpm asrun --env production --read-only` |
136
- | List test data a case could not remove | `pnpm cleanup --env <env>` (`--forget` after deleting it by hand) |
148
+ | Look around a page as a profile | `pnpm explore example-staging /app/home` |
149
+ | Open the last report | `pnpm report` |
150
+ | Read what the last report says about a case | `pnpm heal:evidence --case <case-id> --env staging` |
151
+ | Check the specs | `pnpm spec:lint [case-id]` |
152
+ | List the baselines next to what they must show | `pnpm baselines:review [case-id] [--new]` |
153
+ | Update the baselines of some cases on purpose | `pnpm asrun --env staging --grep <case-id> --update-baselines <case-id>` |
154
+ | List test data that could not be removed | `pnpm cleanup --env staging` |
137
155
  | Watch a run step by step | `pnpm asrun --env staging --ui` |
138
- | Show the current test step as text on the videos | `pnpm asrun --env production --step-text` (also `=1\|true\|yes\|on`; off by default) |
139
- | Open the last report (videos, screenshots, traces) | `pnpm report` |
140
- | Look around a page as a profile (files you download in that window are saved to `~/Downloads`; the path is printed in the terminal / job log) | `pnpm explore <profile-id> /some/path/` |
156
+ | Put the current step on the videos | `pnpm asrun --env production --step-text` |
141
157
 
142
- ### Local web UI
143
- ```bash
144
- pnpm go # install + build + start + open the browser (use this the first time)
145
- pnpm ui # builds the web app and serves http://127.0.0.1:4173
146
- pnpm ui:dev # development: Vite on :5173 with hot reload, API on :4173
147
- ```
148
- One page for the daily work (it only listens on `127.0.0.1`):
158
+ The web UI: `pnpm go` (install, build, start, open), `pnpm ui` (build and serve), `pnpm ui:dev` (development with hot
159
+ reload). Each page covers one job: **Profiles** (login state, sign in, visit, add/duplicate/delete), **Cases** (list,
160
+ editor, form panel, spec view, AI runs, notes-to-cases), **Run tests** (filters, update baselines, production
161
+ confirmation), **Reports** (failure first, then video, screenshots, trace, full Playwright report), **Jobs** (live log
162
+ of every command), **Help** and **AI guide**. See [web-ui.md](./docs/web-ui.md).
149
163
 
150
- | Page | What it does |
151
- |---|---|
152
- | Profiles | Login state of every profile, one row each (same check as `pnpm status`). "Log in" starts `pnpm signin <profile-id>`; log in in the browser window that opens, then press "Done, save the session" on the card in that window's bottom-left corner (the "I've logged in, save the session" button on the job does the same, from the UI). A logged-in profile shows `Login YES`. Every row has a "..." menu: a logged-in profile gets "Verify" (same read-only session check) and "Re-login" (starts `pnpm signin` again), a saved session gets "Logout" (deletes the saved session file, after a confirmation), and every row gets "Duplicate" (adds a second entry with the same settings under a new id; the saved login is not copied) and "Delete" (removes the entry from `profiles.yaml` and deletes its saved login, after a confirmation; screenshot baselines and old reports keep the id in their names and stay on disk). "Visit" (shown for logged-in profiles) opens a logged-in browser window at the profile's `startPath`, or the site root when it is empty (`pnpm explore <profile-id> <startPath>`). While that window is open the button turns into a red "Close browser", which kills the browser process. Files downloaded in that window are saved to `~/Downloads` (an existing file is never overwritten, a ` (1)` is added) and the saved path is printed in the job log. "Add a new profile" has a form (environment, URL, account label, tags, start path) that appends an entry to `profiles.yaml`, plus a guide for what to do next (log in, update the cases, a copyable AI prompt). |
153
- | Cases | List of `cases/**/*.md`, grouped by folder (collapsible), source editor with a rendered preview (both highlight what needs a human: `TODO` in any spelling such as `todo` / `TO-DO` / `to-do` (red for `🚫 TODO [blocking]`, gold otherwise), `🔍 to be confirmed by QA` / `👁️ not observed` / `TBC` / `TBD` in blue, and your `✍️ Answer:` lines in green; the emoji in front is part of the highlighted marker; hover a highlighted word for a short tooltip on what it means and how to resolve it, and click the small X after it to delete the marker, or the whole question block for a `TODO`, in one step, in the editor and in the preview (Undo brings it back; the file is only saved when you save). An `Answer:` gets its X once you typed something, and it clears what you typed), the editor and the preview scroll together, the preview block that holds the caret is shaded, and a click in the preview puts the caret on that line of the editor; `Ctrl/Cmd+S` to save, "New case" has an optional folder field and runs `pnpm case:new`. The **Spec** view is editable too (`Ctrl/Cmd+S` saves; the save is refused when the file changed on disk meanwhile): **Open in editor** opens the spec in the default application of your system for `.spec.ts` files, and **Copy path** copies its absolute path, to paste into a terminal editor such as nvim. Click the file name (or the pencil icon next to it) to rename the case or move it to another folder: the spec and the baseline images move with it and the spec is rewritten to match. **Delete** (type the id to confirm, or click the feather icon to fill it in) removes the case, its spec and its baseline images; both dialogs list exactly what they touch first, and are refused while an AI or test run is in progress. The editor toolbar has a command picker (explore / generate / heal, explore by default), an **Environment** picker (required for every command, no default), an **Attended** box (ticked by default, only for explore), a **Run AI** button (for **Heal spec** it opens a dialog first, see below) and icon buttons to copy the shell command and open the AI guide, for the one-shot AI command of the open case. The Claude / OpenCode switch (Claude by default, remembered in the browser) is in the top bar and applies to every AI command. The **Model** picker beside it chooses the model those commands run with, out of the models that CLI has on this machine (read from `opencode models`, and from Claude Code's own settings, gateway or aliases; searchable, remembered per CLI); without a choice, OpenCode's free built-in model is what runs. The **Thinking** icon beside the picker turns the agent's thinking on for every AI run and lights up while it is on (off by default). **From memo** opens a Markdown editor with its rendered preview beside it for raw test notes: **Make cases** saves them to `.tmp/memos/` and runs `/make-case` one-shot (no questions: defaults for gaps, a `TODO` block for unknown values, no page exploration, existing cases are never overwritten); **Copy /make-case** copies the memo for the full interactive dialogue. When the run ends, the **What to do next** bar shows how many cases are new; **Review** opens a sheet listing each one, with an **Explore** button that opens the case with Explore case selected: a case written from a memo has not been checked against the real pages yet. |
154
- | Run tests | Form for `pnpm asrun`: environment (required, no default), profiles, suite, read-only, case, step text. The **Case** field is a searchable picker: type part of a case id and pick it from the list (only cases that have a spec are offered). The **Cases to run** list shows what the filters select, with an **Update baselines** box per case (`--update-baselines`, unticked by default; asks before it overwrites and shows the new images next to their target cards when the run ends). Asks for confirmation when write cases would run in production. |
155
- | Reports | One entry per run, each with a PRODUCTION / STAGING badge (runs from before environments existed are shown as PRODUCTION). **Results** lists every case x profile with the failure reason on one line (failed first); selecting one shows a red "Why it failed" block (failed step + error message) above the screenshot, video and a folded "Technical details" (call log, stack trace). **Full Playwright report** embeds the standard HTML report (videos and traces work). The latest run opens first. Runs made before summaries existed only have the full report. |
156
- | Help (the **?** icon in the top bar) | "How to start": nine short steps with screenshots, from installing AsRun to reading a report. The screenshots are in `ui/web/public/help/` (retake them when a page changes). |
157
- | AI guide | How each AI step works in the Web UI (which runs the AI tool for you), and the same `/make-case`, `/explore-case`, `/gen-spec`, `/heal-spec` commands for Claude Code and OpenCode in a terminal. |
158
-
159
- The case editor opens in **Single** with the **Panel** full width. The toolbar switches it to **Split** (source on the
160
- left, one pane on the right; drag the divider to resize them, double-click it to reset), or, still in Single, to
161
- **Editor** or **Doc** (the case rendered as a document). In Split the right pane is **Panel** (the default) or **Doc**;
162
- its switch sits at the right end of the toolbar, because that is the pane it controls. **Panel**
163
- shows the case as a form, with what needs a person first: the open questions (the text above each one is what the case
164
- will say, so it is shown in full — accept it to delete the question, or write an answer and the AI folds it in; a click
165
- anywhere else on a question jumps to it in the source editor and flashes the text above with the whole block, while the
166
- answer field and the buttons keep their own behaviour) and the
167
- `🔍 to be confirmed by QA` / `👁️ not observed` markers (click a marker to jump to its line in the source, or confirm it
168
- to remove the marker and keep the sentence; the strip at the top counts them, so a collapsed list still says what
169
- waits). Then the front matter (title, login, suites, writes and its note, environments), the steps (the sentence, its
170
- `[writes]` / `[input]` / `[email]` tags, the `- Ready when:` lines — the signal that the content a step loads is really
171
- there, which the generated test waits for before it looks — and the `- Recipe:` lines of the steps a person does: what
172
- they did in the browser, recorded by `/explore-case --attended`; several lines per step are normal, and they are not
173
- test data — the asserted values are under Values), the `## Expected` and `## Visual checks` bullets (drag the handle, or
174
- Space and the arrow keys, to reorder them with their questions), the `## Values` table (one column per profile; an empty
175
- cell inherits `default`, clearing a cell removes it from that profile), and the `## Targets` cards, read-only: one row
176
- per card, with the fields it is missing and whether no check uses it. Every `[T1]` in a step, in a ready signal or next
177
- to a check is a chip — orange, because it is a reference, with a white fill when the card is ready, amber when it needs
178
- work and red when there is no card — and clicking it (or a Targets row) opens the card in a window: what it is, where it
179
- is, how it is found, what it must not be confused with, and its reference screenshots per profile (click one for the
180
- full size), with **Open in editor** to edit the text. The screenshots are local state under `.explore/` (git-ignored): on a machine
181
- that never ran the exploration the card keeps the path and says the file is not there. The panel's text
182
- fields (a step sentence, its ready signals and recipe lines, a bullet, an answer, the writes note) are small Markdown
183
- editors with the source editor's highlighting and monospace font; they grow with their text and scroll past a height
184
- limit. `## Goal`, `## Notes` and `## Source` are read-only: clicking a paragraph moves the caret of
185
- the source editor to that line and flashes it three times; in **Single** a click there (and every **Open in editor**
186
- button) also switches the pane back to **Editor**, so the line never opens in a pane nobody sees.
187
- Every section collapses, each title carries the icon of what the section holds, and every title has a **?** icon with a
188
- short help card on what it is for and how to work with it. Every edit lands in the same draft as the source editor and
189
- rewrites only the bytes it changes (prose, comments and blank lines stay exactly as they are), so `Ctrl/Cmd+S` saves it
190
- and **Discard** throws the panel edits away too. The model behind it is `lib/case-doc.ts`, the same parser
191
- `pnpm case:check` uses.
192
-
193
- The "Jobs" bar at the bottom (opening it shrinks the page above instead of covering it) shows every command started from the UI with live stdout/stderr and its exit
194
- code; a job can be cancelled there. Only these fixed actions can run (`signin <profile>`, `visit <profile>` and `run` with
195
- checked options) the three AI commands for an existing case and `/make-case` for a memo (`claude -p` / `opencode run`); there is no free shell. Set `ASRUN_UI_PORT` to change the port.
196
-
197
- An AI job's log is read, not skimmed: a timestamp in its own style, one line per tool call (its name, what it touched and
198
- how long it ran), the first lines of what that call returned with the colors a command printed, the agent's thinking
199
- dimmed and italic while **Thinking** is ticked, and what the agent says rendered as Markdown. The log file kept under
200
- `.tmp/ui-jobs/` holds the same text with the timestamps written out. Long output is a preview (six lines, then a count),
201
- so a page snapshot or a big file cannot flood the log. When an AI job finishes, a thin **What to do next** bar appears
202
- under the log; click it (or **Details**) to open its full text in a sheet from the right. The log keeps its full height,
203
- so a short Jobs drawer has one scrollbar in the log, not a second one in the hint.
204
-
205
- The **Model** picker next to the Claude / OpenCode switch chooses the model the AI commands run with. It does not
206
- carry its own list: it reads the models of the CLI that is selected on this machine, so it shows the providers you
207
- already signed in and the models your own configuration names.
208
- - OpenCode: `opencode models`, i.e. every provider in `opencode auth list`, then passed as `--model`. Without a
209
- model, a one-shot `opencode run` uses OpenCode's **built-in free model**, whose daily limit a long day of runs can
210
- use up; the picker preselects the model your OpenCode TUI last used. Set `"model"` in `opencode.jsonc` to stop
211
- depending on this browser's choice.
212
- - Claude Code: there is no command that lists models, so the rows come from what Claude Code itself knows on this
213
- machine (`model`, `modelPicker`, `ANTHROPIC_CUSTOM_MODEL_OPTION`, `ANTHROPIC_DEFAULT_*_MODEL` in its settings and
214
- environment, plus the aliases `sonnet` / `opus` / `haiku` / `fable` / `opusplan`), and, for a gateway set with
215
- `ANTHROPIC_BASE_URL`, from that gateway's `/v1/models` (the request Claude Code makes when its model discovery is
216
- on, with your `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` / `ANTHROPIC_CUSTOM_HEADERS`). A gateway that serves
217
- other models, such as Gemini, shows them too. Cloud providers (Vertex, Bedrock, Foundry) are **not** queried:
218
- Claude Code does not list their models either, so only the configured ones appear there.
219
-
220
- The choice is remembered per CLI in the browser and is shown in the command line of every AI run.
221
-
222
- The **Thinking** icon beside the model picker (a small chip; lit while it is on) passes `--thinking` to OpenCode, so the
223
- model is asked to think and the job log shows it. It is off by default: thinking costs tokens and makes a run slower.
224
- Claude Code has no such flag, so the icon is disabled there; it shows thinking when its model and settings have it on.
225
- Whether anything appears also depends on the model: one that does not return its reasoning shows none.
226
-
227
- ### Checking what a test actually did
228
- Every run records a video and a trace per test. A new browser tab gets its own recording; the **Reports** page joins
229
- them in order into one video per test (`reports/<run>/videos/`, needs ffmpeg, see Quick start). The full Playwright report
230
- still lists the recordings separately. Add `--step-text`
231
- to `pnpm asrun` to overlay the current step's title on the videos; without it the videos are plain.
232
- In `pnpm report`, open the test: the steps are named after the steps in the case document. The trace shows each action with
233
- the page before and after it, plus every network request. `pnpm asrun` also prints the file paths of the
234
- videos and traces when it finishes; open a trace directly with `pnpm exec playwright show-trace <trace.zip>`.
235
- Recordings contain real production pages, so do not share them.
236
-
237
- ### Logging in
238
- `pnpm signin <profile-id>` opens a normal browser with a small card in its bottom-left corner. Log in yourself,
239
- including 2FA, wait for the logged-in page, then press **Done, save the session** on that card (or press Enter in the
240
- terminal, which does the same). The session is saved to `.auth/<profile-id>.json` and verified. Nothing about 2FA is
241
- automated.
242
-
243
- Login itself is not tested. Before every run, `pnpm asrun` checks the session of each profile of the chosen
244
- environment. If a profile has no saved session, or the session expired (the check page redirects to `/login`), the
245
- tests of that profile fail right away (no retries) and the run prints:
246
- ```
247
- Not logged in (tests for these profiles will FAIL):
248
- example-production expired -> pnpm signin example-production
249
- Or skip them for this run: pnpm asrun --env production --skip-profile example-production
250
- ```
251
- Either log in (`pnpm signin <profile-id>`) or leave the profile out with `--skip-profile`. Profiles that are logged
252
- in are tested as usual. If a session expires in the middle of a run, the failed test is marked
253
- "session expired" in the output.
254
-
255
- Sessions are as sensitive as passwords: `.auth/` (one file per profile) is git-ignored, never share or commit it.
256
-
257
- ## From a case document to a test (AI workflow)
258
-
259
- 0. **(Optional) start from raw notes.** If you only have a memo or a list of things to check, run
260
- `/make-case path/to/notes.md` (or paste the text after the command). The agent normalizes the text into steps in
261
- the project's artifact language (English by default), marks steps that write data (`[writes]`) or send email
262
- (`[email]`), keeps your original lines in a `## Source` table, lists the gaps with a suggested default for each,
263
- and only then writes `cases/<id>.md`. It never writes specs and
264
- never invents expected values (unknown ones become a default sentence with a `TODO` block under it). It never opens a
265
- browser: what the pages really show is found in step 2. `pnpm case:check` validates the front matter of every case.
266
- 1. **Write the case.** `pnpm case:new header-logo` creates `cases/header-logo.md` from a template; `pnpm case:new documents/header-logo` puts it in the
267
- group folder `cases/documents/` (created on demand, one level only). The id of a case is its path without `.md`, its spec is
268
- `tests/documents/header-logo.spec.ts`, and its expected values are in its own `## Values` block.
269
- Fill in steps, what is expected for every profile, and the exact values the spec asserts under Values, one entry per
270
- profile id (`default` applies to all). Plain language is fine for the checks. Write it in the project's artifact
271
- language (`AGENTS.md` §2; English by default); if a case is in another language, `/gen-spec` translates the case
272
- file to the artifact language first (and tells you).
273
- 2. **(Optional) explore the case.** In OpenCode or Claude Code run `/explore-case header-logo production` (the case id is
274
- required; the environment is optional: `production` or `staging`). This is the only step where a person and the
275
- agent work together. The agent opens the real pages of the profiles that are logged in and only **reads** them: it
276
- never creates, edits, publishes, shares or deletes anything, in staging or in production. It fixes wording in the
277
- case file, writes what the pages show into it (marked `observed, 🔍 to be confirmed by QA`), and turns what it cannot
278
- know into `🚫 TODO [blocking]` / `❓ TODO [non-blocking]` questions (every marker starts with its emoji). Every question is a block
279
- under a sentence that is the agent's suggestion, written so that it can stand in the case as it is:
280
- ```
281
- - Clicking "Edit Draft" opens the document editor, in the same tab or in a new tab.
282
- - ❓ TODO [non-blocking]: Same tab or new tab? (👁️ not observed)
283
- - ✍️ Answer:
284
- ```
285
- It edits only the case file. **To accept the suggestion, delete the whole block** (the `TODO` line and the
286
- `Answer:` line). **To give your own answer, write it after `Answer:`** (you may delete the rest of the block).
287
- Then run the command again. It ends with `RESULT: ready` or `RESULT: needs-input`. Under the output of an Explore run,
288
- a one-line **What to do next** bar shows what is left: the open questions (a red badge when one must be answered),
289
- what the exploration marked for someone to confirm, and **Generate spec**, which opens the case with that command
290
- selected. **Details** (or **Answer** when the agent waits) opens a sheet from the right with the question list, the
291
- answer box and **Use the suggested defaults**, the confirmations and the generate action. A run that ends with
292
- `needs-input` shows an amber "Needs your input" badge; sending answers (or "Use the suggested defaults", which
293
- accepts every suggestion) continues the same agent conversation with the same permissions, folds them into the case
294
- file and ends with a new `RESULT:`. Other commands with open questions show the answer box in the same sheet. A run
295
- can be continued once per turn, and only while the UI server has not been restarted; after a restart, answer in the
296
- case file and run the command again.
297
-
298
- **Attended exploration (`--attended`, or the "Attended" box on the Cases page, ticked by default).** Use it for a case that creates or
299
- edits data, or needs a choice (an existing document, items). Stay at the computer: the browser window opens and the
300
- agent waits while you do those steps. The window shows a small card in the bottom-left corner that says what to do (drag it by its title bar or minimize it if it is in the way), with **Done** (hand
301
- control back) and **Cannot do this**; the agent also continues by itself when the finish condition it named shows up
302
- on the page (for example a "Saved" message), and the card turns green when control is back. It waits up to 10
303
- minutes per step. Delete the `[asrun]` items you created before you press Done on the last step. What you did is
304
- saved in the case under the step as **general recipe lines** (`- Recipe (phase): ...`): positions instead of names
305
- ("select items #1 to #3"), `[asrun]` names for created data, control labels but no selectors. Without `--attended`
306
- the agent never waits: a step that needs a person becomes a `TODO [blocking]`.
307
- 3. **Log in the profiles the case needs** (`pnpm status`, `pnpm signin ...`). Skip for `login: none` cases.
308
- 4. **Generate.** In OpenCode or Claude Code run `/gen-spec header-logo staging` (the environment is required; there is
309
- no default). The agent replays the case on the real pages with your saved sessions (it never writes data), writes `tests/header-logo.spec.ts`, fills the
310
- case's `## Values` for the profiles it explored and checks the files (`pnpm typecheck`, `pnpm case:check`). It does not run the
311
- spec and does not create screenshot baselines: running specs is a separate step (6). It does not stop to ask: what the
312
- page answers is written into the case as observed (QA confirms it), and what is still unknown stays in the case as a
313
- `TODO`. It ends with `RESULT: ready` or `RESULT: needs-input`. A case with a `[writes]` or `[input]` step that has no
314
- `- Recipe:` line under it is not generated: it stops with `needs-input` and asks you to run step 2 with
315
- `--attended`.
316
- If the spec already exists, the command leaves it alone and says so. Add `--force` (`/gen-spec header-logo staging
317
- --force`) to regenerate it from the case and overwrite it (the case's `## Values` and baseline images are kept; the
318
- report lists the ones that may be out of date). In the web UI, **Run** on a case that has a spec
319
- asks first: "Overwrite the spec?". The server refuses to start `gen-spec` on an existing spec without that choice.
320
- When the run ends, **What to do next** offers **Open Run tests** with the case already selected, for step 6.
321
- 5. **Review what the AI produced.** Check: the spec reads like your case, and the `## Values` block of the case has the
322
- right values for each profile (you normally don't edit these by hand, but you can).
323
- 6. **Run it:** `pnpm asrun --env <env> --grep <case-id>`. If the spec has visual checks, add `--update-snapshots`
324
- the first time, then open the new images in `tests/<case>.spec.ts-snapshots/` and confirm each is the correct logo
325
- for that profile. Baselines are the source of truth for pixel checks. Then run all profiles and `pnpm report`.
326
-
327
- When the product changes and a test fails, run `/heal-spec <case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]`
328
- (or **Heal spec** in the Cases toolbar, or **Heal this failure** on a failed result on the Reports page). The agent
329
- decides whether the test needs updating or whether it found a real problem, and never edits an expectation just to
330
- make a failure disappear. How it works:
331
- - The environment is always required. Only the profiles that are logged in there are run (`pnpm asrun --logged-in`
332
- leaves the others out instead of failing them).
333
- - The web UI first opens a dialog: the profiles to heal (every logged-in profile ticked) and an optional text box for what
334
- you want healed or any evidence and clues. The text is stored in `.tmp/hints/` (git-ignored) and passed as `--hints`.
335
- Clues guide the diagnosis; they never lower the bar for a product finding.
336
- - The agent reads the existing report first (`pnpm heal:evidence --case <id> --env <env>`: status, failed step, error,
337
- call log and screenshots per profile, and whether the report is fresh or stale) and runs the case again only when
338
- there is no report, it is stale, it does not explain the failure, or a clue has to be checked. After it edits the
339
- spec it runs it once to confirm.
340
-
341
- ### Targets, readiness and downloads (why generated specs can be trusted)
342
- Three rules keep an AI-written spec from passing or failing for the wrong reason. They hold in exploring, generating and
343
- healing (`AGENTS.md`, `<framework>/.claude/skills/exploring-cases/references/assertion-targets.md` and `readiness.md`).
344
- - **Targets.** A check on a logo, image, icon, file, PDF, screenshot or repeated text points to a card under `## Targets`
345
- in the case (`[T1]`): what it is (and what it is not), where it is, how it is identified (never `.first()` or a bare
346
- `img`), when it is ready, and a highlighted reference screenshot under `.explore/targets/`. When the agent is in doubt
347
- it does not pick: it writes a `🚫 TODO [blocking]` that asks for the exact image, text, place or UI, with the
348
- candidates listed. Answer it by editing the case (or, in an attended run, by outlining the target in the browser).
349
- - **Readiness.** After an action that loads something, the key requests must finish and the content must be visible before
350
- anything is read. Specs use `actAndWait` (`@asrunjs/asrun/ready`); "missing" and "empty" checks need a positive ready signal.
351
- Healing treats a "missing / empty / timeout" failure as a readiness problem first.
352
- - **Downloads.** `verifyDownload` (`@asrunjs/asrun/downloads`) checks that the download starts, finishes, the file exists and the
353
- file is valid for its kind; any failure or timeout fails the whole step with a message that names the stage.
354
- - **PDFs are read like a person reads them.** `expectPdf` (`@asrunjs/asrun/pdf`) renders the pages to images (a logo region must not
355
- be blank and must match the profile's baseline, 2% tolerance) and extracts the text (for example, no other profile's name).
356
- Every page is checked unless the case limits the pages in strong words ("only", "exactly", "must"); "should",
357
- "always", "usually" or no wording mean every page.
358
- `pnpm pdf:render <file.pdf> --out .explore/targets/<case-id>/<name> --corner top-left` lets you (or the exploring agent)
359
- look at the rendered pages.
360
- - **Checks you can run yourself:** `pnpm spec:lint` (fails on screenshots without a target, `.first()` on images, sleeps,
361
- `networkidle`, raw download events, a `%PDF` check on its own), `pnpm baselines:review` (opens nothing; lists every
362
- baseline with the card that says what it must show, so you can look at each new image), and `pnpm case:check`
363
- (fails on an incomplete card, warns on an image check with no card).
364
-
365
- ### Updating baselines on purpose
366
- If a profile really changed its logo, or a baseline is wrong: fix the spec first (the check must look at the element
367
- its target card names; a wrong locator just makes a new wrong image), then
368
- `pnpm asrun --env <env> --profile <id> --grep <case> --update-baselines <case-id>[,<case-id>]`, then look at the new
369
- image before keeping it (`pnpm baselines:review <case-id>`).
370
- - `--update-baselines` names the cases whose images are overwritten (`--update-snapshots=all`). They must be among the
371
- cases the filters run. Playwright has one global switch, so they run as a second pass with their own report
372
- (`<run-id>-baselines`); every other case is compared with its baselines as usual.
373
- - Without `--update-baselines`, `--update-snapshots` still works as before, but it rewrites every image of the run.
374
- - In the web UI, every case in **Cases to run** has an **Update baselines** box (nothing is ticked by default). A
375
- confirm dialog follows, and when the run ends the job output shows each rewritten image next to the target card that
376
- says what it must show.
377
- - Not part of exploring, generating or healing: an agent never updates a baseline.
378
-
379
- ### Suites
380
- Each case lists its suites in the front matter: `suites: [smoke, regression]` (any lowercase words). A case runs in
381
- every suite it lists. The generated spec tags itself `@<suite>` for each one, which:
382
- - shows as badges next to the test in the terminal and in `pnpm report`, and can be typed into the report's
383
- search box (`@smoke`) to filter there;
384
- - drives `pnpm asrun --suite <name>`, which can be combined with `--profile` and `--grep`. A case that is in two
385
- of the chosen suites still runs once.
386
-
387
- An unknown suite is rejected with the list of suites the cases use. The old `priority: <level>` is read as
388
- `suites: [<level>]`.
389
-
390
- ### Other front matter
391
- - `writes: [staging]` or `writes: [staging, production]` (omit it for a read-only case): the environments in which the
392
- case may create or change data of the test accounts (and cleans it up). It then needs
393
- `writes-note: <what it creates or changes>`. In an environment that is not listed, the case is **skipped with a
394
- notice**, not failed. The old `writes: test-data` is refused with this hint.
395
- - `envs: [production, staging]` (optional): the case runs only in these environments; in others it is skipped. Every
396
- environment in `writes` must also be in `envs`.
397
-
398
- Cases that write data get a red **WRITES** badge in the web UI and the report, and an `@writes` tag.
399
-
400
- ## What a case may and may not do
401
- Tests run against production and staging. By default a case is read-only: look, read, compare. No creating,
402
- editing, deleting, submitting or accepting anything.
403
-
404
- A case may write only in the environments listed in its `writes` front matter (elsewhere it is skipped with a notice),
405
- and only data of the test account (named with the prefix `[asrun]`), which it must clean up.
406
- **AI agents never write data while they explore or generate**, in either environment: a person does those steps, in
407
- the browser, in an attended exploration (step 2 above).
408
- - **Staging:** no confirmation needed.
409
- - **Production (and any environment that is not staging)**, for a case that lists it under `writes`: a person confirms every run. `pnpm asrun --env production` lists
410
- the write cases and asks you to type `production`. The web UI shows a confirm dialog. Unattended runs use
411
- `--allow-writes=production`; AI agents never do. `--read-only` leaves the write cases out. The permission is also
412
- enforced inside the test itself, so running `playwright test` directly cannot bypass it.
413
- - **Cleanup:** data is named `[asrun] ...`, registered with the `records` fixture and removed after the test, also
414
- after a failure. Whatever could not be removed is shown in red in the report and listed by
415
- `pnpm cleanup --env <env>`; delete it by hand, then `pnpm cleanup --env <env> --forget`.
416
- - **No retries, one at a time:** write cases are never retried, and run one at a time per profile.
417
- - **Email:** never sent from production. In other environments only if the profile has `email: true` in `profiles.yaml`.
418
-
419
- ## Author
420
- Created by Zoro Chen.
421
-
422
- ## Layout
423
-
424
- **In a project that uses the framework** (everything here is committed except the ignored folders):
164
+ ## Safety
425
165
 
426
- ```
427
- profiles.yaml one entry per environment: URL, account label, tags, environment (no expected values)
428
- cases/ human-written cases (+ _template.md); optional group folders: cases/<group>/<name>.md
429
- tests/ generated specs (same folders as cases/), and *-snapshots/ baseline images per profile
430
- tsconfig.json plain compiler options: specs import the framework by package name (`@asrunjs/asrun/fixtures`)
431
- playwright.config.ts re-exports node_modules/@asrunjs/asrun/dist/playwright.config.js
432
- AGENTS.md the project rules: they point at the framework baseline, and §2 says what differs (narrows or replaces)
433
- .claude/commands/ /make-case, /explore-case, /gen-spec, /heal-spec: thin shells that load the framework skill (Claude Code)
434
- .opencode/commands/ the same commands for OpenCode (it does not read `.claude/commands`)
435
- .claude/skills/ optional project overrides of a skill: <skill>.md, read by the command and the skill (not a discovered skill)
436
- .mcp.json, opencode.jsonc the playwright-test MCP server, for Claude Code and OpenCode
437
- .env ASRUN_ROOT (which workspace the commands use) and local settings (git-ignored)
438
- node_modules/@asrunjs/asrun/ the framework (installed dependency; never edited)
439
- .auth/ saved logins, one per profile (git-ignored)
440
- reports/ one folder per run; reports/latest points to the newest (git-ignored)
441
- ```
166
+ Tests run against real sites, so the defaults are strict:
442
167
 
443
- **In the framework repository** (no workspace content at the root: no `profiles.yaml`, `cases/`, `tests/` or reports):
168
+ - **Read-only unless the case says otherwise.** A case may create, edit or delete data only in the environments it
169
+ lists under `writes`, and only data of the test accounts, named `[asrun] ...`, which it must clean up. Elsewhere the
170
+ case is skipped with a notice, not failed.
171
+ - **A person confirms non-staging writes on every run** by typing the environment name, or in a dialog in the UI.
172
+ `--read-only` leaves those cases out; `--allow-writes=<env>` is for unattended runs, and AI agents never use it.
173
+ - **No email from production.** Elsewhere only when the profile has `email: true`.
174
+ - **AI agents never write while exploring or generating.** A step that writes or needs a choice is done by a person in
175
+ the browser, in an attended exploration. Logins are always a person's job, 2FA included.
176
+ - **A failing baseline is a person's decision.** An agent never updates a baseline to make a difference disappear.
444
177
 
445
- ```
446
- lib/ fixtures, session guard, image assertions, case values (imported as asrun/<module>)
447
- scripts/ the pnpm commands above, plus init (workspace files) and doctor (checks)
448
- templates/ what `pnpm asrun init` writes into a project
449
- ui/ local web UI: server/ (Hono API, job runner), web/ (React + Vite), shared/ (types)
450
- docs/ static docs; theme-guide.html = the Inkstone theme (designed and created by Zoro Chen): colors, buttons, icons of the web UI (open in a browser, or via the "Inkstone theme" link in the UI's top bar)
451
- examples/demo/ a committed workspace for developing the framework (public demo site; .auth/ and reports stay local)
452
- .claude/ skills + slash commands for Claude Code; the skills are shared with OpenCode
453
- .opencode/commands/ the same commands for OpenCode (it does not read `.claude/commands`)
454
- AGENTS.md the product baseline rules (OpenCode reads it); CLAUDE.md imports it for Claude Code
455
- docs/agents/ framework-dev.md: how to change asrun itself, never used by a project
456
- opencode.jsonc MCP config for OpenCode; .mcp.json is the same server for Claude Code
457
- ```
178
+ The rules that enforce this are `F1`–`F10` in [AGENTS.md](./AGENTS.md); a project can tighten or replace the last
179
+ four, never the first six.
458
180
 
459
- ## Developing the framework
181
+ ## Documentation
460
182
 
461
- `pnpm go`, `pnpm asrun`, `pnpm ui` work here too: without a workspace they use `examples/demo`. `init` writes the
462
- framework's `.env` for that (`ASRUN_ROOT=examples/demo`), so the commands need no environment variable:
183
+ | I want to... | Read |
184
+ |---|---|
185
+ | Install, update, requirements, workspace location | [docs/install.md](./docs/install.md) |
186
+ | `profiles.yaml`, environments, logins, sessions | [docs/profiles.md](./docs/profiles.md) |
187
+ | Write or review a case | [docs/cases.md](./docs/cases.md) |
188
+ | The four AI commands in depth | [docs/ai-workflow.md](./docs/ai-workflow.md) |
189
+ | Every command and flag, reports, baselines | [docs/commands.md](./docs/commands.md) |
190
+ | The local web UI, page by page | [docs/web-ui.md](./docs/web-ui.md) |
191
+ | An error message | [docs/troubleshooting.md](./docs/troubleshooting.md) |
192
+ | Change AsRun itself | [docs/agents/framework-dev.md](./docs/agents/framework-dev.md) |
193
+ | The case document contract, for AI agents | [docs/agents/case-format.md](./docs/agents/case-format.md) |
194
+
195
+ ## Maintainers
196
+
197
+ This section is for the people who maintain AsRun; the repository is private, so the instructions below assume
198
+ access to it. It is developed against `examples/demo`, a committed workspace whose site is a local Docker stack
199
+ (Conduit), so nothing runs against a public site.
463
200
 
464
201
  ```bash
465
- pnpm asrun init --root examples/demo # creates the demo workspace and the framework-root .env
202
+ pnpm install
203
+ pnpm asrun init --root examples/demo
204
+ pnpm build # the demo's specs import the package, which resolves to dist/
205
+ pnpm demo:up # Docker Desktop must run
206
+ pnpm demo:seed
207
+ pnpm signin conduit-a # password in examples/demo/README.md
466
208
  pnpm asrun --env staging --read-only
209
+ pnpm ui
467
210
  ```
468
211
 
469
- The demo is a workspace of its own as well:
470
-
471
- ```bash
472
- cd examples/demo && ASRUN_ROOT=. pnpm asrun doctor
473
- ```
212
+ `pnpm demo:reset` returns the database to the state the cases expect (logins must be redone). Before a pull request:
213
+ `pnpm typecheck`, `pnpm build`, `pnpm ui:build`, and the checks above on the demo. A change to `templates/` or the
214
+ shipped docs bumps `version` in `package.json`.
474
215
 
475
- `ASRUN_ROOT` still wins over `.env` (a real environment variable always does), so one checkout can be pointed at
476
- another project for a single command:
216
+ Framework layout, the two agent CLIs, the UI theme and copy rules, and where documentation belongs:
217
+ [docs/agents/framework-dev.md](./docs/agents/framework-dev.md).
477
218
 
478
- ```bash
479
- ASRUN_ROOT=../my-real-project pnpm asrun --env staging --read-only
480
- ```
219
+ ## License
481
220
 
482
- Two paths keep the two halves apart: `ROOT` (`lib/paths.ts`) is the workspace, `FRAMEWORK_ROOT` is this folder. The
483
- package is compiled to `dist/` (`pnpm build`) and the web UI to `ui/dist` (`pnpm ui`); both are shipped by `prepack`.
484
- Nothing writes into `FRAMEWORK_ROOT` at run time. After changing `lib/`, run `pnpm build` before exercising the demo's
485
- specs (they import the package by name, which resolves to `dist/`). After changing `templates/`, bump `version` in
486
- `package.json`; `pnpm asrun doctor` in a project then shows that it is behind.
487
-
488
- ## How profiles become tests
489
- Every entry in `profiles.yaml` becomes a Playwright project (name = profile id) with its own `baseURL` and saved login.
490
- A spec is written once and runs once per profile; the `profile` fixture exposes that profile's `id`, `env`, `baseURL`,
491
- `account`, `tags` and `email`. The expected values come from the case document, through
492
- `requireValues(caseId, profile)` (its `## Values` block, `default` merged with the profile's own entry). Adding a
493
- profile = add a YAML entry (by hand or with the form on the Profiles page) + `pnpm signin <id>`, then add the values
494
- for it to the cases (see the guide on the Profiles page).
495
-
496
- ## Troubleshooting
497
- - `Profile "x" is not logged in`: `pnpm signin <profile-id>`, or run with `--skip-profile <profile-id>`.
498
- - Screenshot mismatch: open the report; the diff view shows expected / actual / diff. If the change is
499
- intended, update the baseline as above.
500
- - Report empty or missing: run `pnpm asrun --env <env>` first.
501
- - `Choose the environment to test`: there is no default; add `--env production` or `--env staging`.
502
- - An AI command says the `playwright-test` MCP server is missing: run `pnpm asrun doctor`. OpenCode reads only
503
- `opencode.jsonc` and Claude Code only `.mcp.json`; `pnpm asrun init` writes both, and `opencode mcp list` /
504
- `claude mcp list` shows whether the server started (`failed: ... path specified` means the `playwright` path in
505
- that file does not resolve: `pnpm asrun init --force <file>`).
506
- - A project rule does nothing: `AGENTS.md` §2 needs the rule id and an `Effect` of `narrows` or `replaces`; `F1`..`F6`
507
- cannot be changed. `pnpm asrun doctor` names a wrong id, a missing `Effect` or a non-negotiable rule.
508
- - A skill override does nothing: it is `.claude/skills/<skill>.md` (the skill name, for example `generating-specs`,
509
- not the command name) and it is not discovered as a skill: the command and the skill read it. `doctor` lists the
510
- overrides it sees.
511
- - `pnpm asrun init --force AGENTS.md` is refused: `AGENTS.md`, `CLAUDE.md` and `.claude/skills/**` belong to the
512
- project. Add `--reset-rules` to replace them with the template, and expect to lose the project's rules.
221
+ MIT. Created by Zoro Chen.