@asrunjs/asrun 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/explore-case.md +11 -6
- package/.claude/commands/gen-spec.md +10 -6
- package/.claude/commands/heal-spec.md +13 -9
- package/.claude/commands/make-case.md +1 -1
- package/.claude/skills/exploring-cases/SKILL.md +65 -31
- package/.claude/skills/exploring-cases/references/assertion-targets.md +8 -4
- package/.claude/skills/exploring-cases/references/case-rules.md +110 -0
- package/.claude/skills/exploring-cases/references/handoff-card.js +73 -11
- package/.claude/skills/exploring-cases/references/open-questions.md +58 -6
- package/.claude/skills/exploring-cases/references/readiness.md +4 -3
- package/.claude/skills/exploring-cases/references/tools.md +117 -0
- package/.claude/skills/exploring-cases/references/user-handoff.md +46 -17
- package/.claude/skills/generating-specs/SKILL.md +71 -33
- package/.claude/skills/generating-specs/references/conventions.md +4 -0
- package/.claude/skills/making-cases/SKILL.md +14 -7
- package/.claude/skills/making-cases/references/gaps.md +7 -4
- package/.claude/skills/making-cases/references/normalizing.md +1 -1
- package/AGENTS.md +51 -11
- package/README.md +4 -3
- package/bin/asrun.mjs +1 -0
- package/dist/lib/case-doc.d.ts +41 -4
- package/dist/lib/case-doc.js +109 -17
- package/dist/scripts/check-cases.js +89 -1
- package/dist/scripts/doctor.js +50 -3
- package/dist/scripts/markers-confirm.d.ts +1 -0
- package/dist/scripts/markers-confirm.js +139 -0
- package/dist/ui/server/actions.js +50 -16
- package/dist/ui/server/app.js +7 -2
- package/dist/ui/server/job-store.d.ts +60 -0
- package/dist/ui/server/job-store.js +198 -0
- package/dist/ui/server/jobs.d.ts +46 -4
- package/dist/ui/server/jobs.js +361 -59
- package/dist/ui/server/process-info.d.ts +17 -0
- package/dist/ui/server/process-info.js +126 -0
- package/dist/ui/server/result-state.d.ts +10 -0
- package/dist/ui/server/result-state.js +106 -0
- package/dist/ui/shared/ai-commands.d.ts +4 -1
- package/dist/ui/shared/ai-commands.js +14 -4
- package/dist/ui/shared/types.d.ts +17 -2
- package/docs/agents/case-format.md +39 -5
- package/docs/ai-workflow.md +47 -15
- package/docs/cases.md +28 -4
- package/docs/commands.md +6 -1
- package/docs/troubleshooting.md +40 -0
- package/docs/web-ui.md +29 -6
- package/package.json +2 -1
- package/templates/.claude/commands/explore-case.md +11 -6
- package/templates/.claude/commands/gen-spec.md +10 -5
- package/templates/.claude/commands/heal-spec.md +11 -7
- package/templates/.claude/commands/make-case.md +1 -1
- package/templates/AGENTS.md +9 -6
- package/templates/cases/_template.md +7 -0
- package/templates/package.json +1 -0
- package/ui/dist/assets/{dist-B7HeglDd.js → dist-2msWI-5M.js} +1 -1
- package/ui/dist/assets/{dist-DfBpjgZT.js → dist-4U_gPHXf.js} +1 -1
- package/ui/dist/assets/{dist-B9_GXR2G.js → dist-BMfJatHo.js} +1 -1
- package/ui/dist/assets/{dist-Bb36rNUw.js → dist-BVvHVJ3N.js} +1 -1
- package/ui/dist/assets/{dist-WvvCZeeu.js → dist-BmWaNjqO.js} +1 -1
- package/ui/dist/assets/{dist-G9KNeh6u.js → dist-BwNz_xW2.js} +1 -1
- package/ui/dist/assets/{dist-GvRUDL-2.js → dist-J36xayOv.js} +1 -1
- package/ui/dist/assets/{dist-Dqc_tfzC.js → dist-e-KhTWHJ.js} +1 -1
- package/ui/dist/assets/{dist-DevASegc.js → dist-sfZKBSMm.js} +1 -1
- package/ui/dist/assets/{dist-Dq4Vnw5N.js → dist-wjB7g6OJ.js} +1 -1
- package/ui/dist/assets/{index-BJ5D2mFK.js → index-CGbRJT3b.js} +134 -132
- package/ui/dist/assets/index-CI0i0LWc.css +2 -0
- package/ui/dist/index.html +2 -2
- package/ui/dist/assets/index-BXY-eKBn.css +0 -2
package/docs/cases.md
CHANGED
|
@@ -31,6 +31,9 @@ suites: [smoke, regression]
|
|
|
31
31
|
## Goal
|
|
32
32
|
Why this case exists, in one or two sentences.
|
|
33
33
|
|
|
34
|
+
## Rules
|
|
35
|
+
- Before step 1: dismiss the cookie banner; it covers the header.
|
|
36
|
+
|
|
34
37
|
## Steps
|
|
35
38
|
1. Open `/`.
|
|
36
39
|
- Ready when: the header has rendered its menu.
|
|
@@ -63,8 +66,8 @@ example-staging:
|
|
|
63
66
|
Anything a reviewer should know.
|
|
64
67
|
````
|
|
65
68
|
|
|
66
|
-
The headings are fixed English tokens (`Goal`, `Steps`, `Expected`, `Values`, `Visual checks`, `Targets`,
|
|
67
|
-
and `Source` when the case was made from raw notes). A project language never translates them.
|
|
69
|
+
The headings are fixed English tokens (`Goal`, `Rules`, `Steps`, `Expected`, `Values`, `Visual checks`, `Targets`,
|
|
70
|
+
`Notes`, and `Source` when the case was made from raw notes). A project language never translates them.
|
|
68
71
|
|
|
69
72
|
## Front matter
|
|
70
73
|
|
|
@@ -80,6 +83,21 @@ and `Source` when the case was made from raw notes). A project language never tr
|
|
|
80
83
|
|
|
81
84
|
`priority: <level>` is the old name of `suites`; it is read as `suites: [<level>]`. Do not use both.
|
|
82
85
|
|
|
86
|
+
## Rules
|
|
87
|
+
|
|
88
|
+
Optional, case-local instructions for the AI agents that explore, generate and heal this case: **how** to run it, never
|
|
89
|
+
**what it proves**. One bullet per rule. A bullet may start with `Step 3:`, `Before step 3:` or `After step 3:` to tie
|
|
90
|
+
it to that step; `pnpm case:check` fails when the step does not exist. A rule without a prefix applies to the whole case.
|
|
91
|
+
|
|
92
|
+
- Use it for a page quirk, a mandatory read-only preparation (dismiss a cookie banner), a control to avoid, or a
|
|
93
|
+
waiting/attention hint.
|
|
94
|
+
- Do not put an expectation, a literal or a target here: those stay in `## Expected`, `## Values` and `## Targets`.
|
|
95
|
+
- A rule never allows creating, changing or deleting data, sending email or doing a person's step. Those stay tagged
|
|
96
|
+
steps with a recipe.
|
|
97
|
+
- A rule that conflicts with the rest of the case is a blocking question, not an override. `/explore-case` re-checks
|
|
98
|
+
each rule against the page and never deletes one silently.
|
|
99
|
+
- The framework baseline's `R1`–`R4` define exactly what a rule may say.
|
|
100
|
+
|
|
83
101
|
## Steps
|
|
84
102
|
|
|
85
103
|
Numbered lines. Keep them short and in plain language: the spec uses each step's text verbatim in `test.step`, so the
|
|
@@ -97,8 +115,13 @@ report and the trace read like the case.
|
|
|
97
115
|
generated: `/gen-spec` stops with `needs-input`.
|
|
98
116
|
- **Open questions** go in a `🚫 TODO [blocking]` or `❓ TODO [non-blocking]` block under the sentence they are about,
|
|
99
117
|
with the agent's suggestion written above and a last line `- ✍️ Answer:`. Accepting a question means deleting the
|
|
100
|
-
whole block; answering means writing after `Answer:`.
|
|
101
|
-
|
|
118
|
+
whole block; answering means writing after `Answer:`. Hang the block under the sentence that makes the claim, not
|
|
119
|
+
under a broader one above it, and one claim gets one block.
|
|
120
|
+
- **What the AI saw** is marked by what happens if it is wrong: `(seen on <profile>)` is a plain note the run checks by
|
|
121
|
+
itself, and `(observed on <profile>, 🔍 to be confirmed: <what a person must decide>)` is a line that would pass even
|
|
122
|
+
if it were wrong, so a person settles it. A marker may name a screenshot (`evidence: .explore/.../x.png`) so nobody
|
|
123
|
+
has to open the page again. Never put a marker inside a TODO block, on the sentence a block hangs under, or inside
|
|
124
|
+
the `## Values` yaml: one claim carries one signal. `pnpm case:check` fails on the first two.
|
|
102
125
|
|
|
103
126
|
## Expected
|
|
104
127
|
|
|
@@ -174,6 +197,7 @@ It reads every case (or the ids you pass) and prints one line per case. It **fai
|
|
|
174
197
|
- a target card without its required fields (`What`, `Where`, `Identified by`), or a reference image outside
|
|
175
198
|
`.explore/targets/<case-id>/`;
|
|
176
199
|
- a `[T<n>]` reference with no card;
|
|
200
|
+
- a `## Rules` bullet that names `Step <n>`, `Before step <n>` or `After step <n>` when the case has no such step;
|
|
177
201
|
- an invalid `## Values` block (bad YAML, an unknown profile, a non-map entry).
|
|
178
202
|
|
|
179
203
|
It **warns** (usable, but something to settle) on: a reference image missing on this machine (`.explore/` is
|
package/docs/commands.md
CHANGED
|
@@ -21,6 +21,7 @@ thin wrappers, so `pnpm asrun`, `pnpm signin` and the rest work as below. The sa
|
|
|
21
21
|
| Read what the last report says about a case | `pnpm heal:evidence --case <case-id> --env <env>` |
|
|
22
22
|
| List or forget test data that could not be removed | `pnpm cleanup --env <env> [--forget]` |
|
|
23
23
|
| Create or validate a case | `pnpm case:new <id>`, `pnpm case:check [case-id]` |
|
|
24
|
+
| Settle the "to be confirmed" lines of a case from a passing run | `pnpm markers:confirm --case <case-id> --env <env> [--dry-run]` |
|
|
24
25
|
| Check a spec | `pnpm spec:lint [case-id]`, `pnpm baselines:review [case-id] [--new]` |
|
|
25
26
|
| Look at a PDF | `pnpm pdf:render <file.pdf> --out <dir> [--pages all] [--corner top-left]` |
|
|
26
27
|
| Type-check the workspace | `pnpm typecheck` |
|
|
@@ -125,7 +126,11 @@ pnpm pdf:render report.pdf --out .explore/targets/header-logo --corner top-left
|
|
|
125
126
|
```
|
|
126
127
|
|
|
127
128
|
- `case:new` creates the case from the template. One optional group folder, created on demand.
|
|
128
|
-
- `case:check` validates front matter, values and target cards;
|
|
129
|
+
- `case:check` validates front matter, values and target cards; it fails on a marker inside a question block, a marker
|
|
130
|
+
on the line a question hangs under, and a marker inside the `## Values` yaml. See [cases.md](./cases.md).
|
|
131
|
+
- `markers:confirm` reads the newest run report (or `--report <run-id|file>`) and removes the `🔍 to be confirmed`
|
|
132
|
+
markers whose line the spec carries as `// unconfirmed: ...`, but only when every test of the case passed. It answers
|
|
133
|
+
"does the page still say this?", never "is this the right expectation".
|
|
129
134
|
- `spec:lint` is a static check of the specs: it fails on sleeps, `networkidle`, image checks without a target,
|
|
130
135
|
`.first()` on images, raw download events, a magic-number-only PDF check. `pnpm spec:lint <case-id>` limits it.
|
|
131
136
|
- `baselines:review` opens nothing; it lists every baseline next to the target card that says what it must show.
|
package/docs/troubleshooting.md
CHANGED
|
@@ -57,6 +57,11 @@ and regenerate the spec.
|
|
|
57
57
|
`Identified by`), refers to a reference image outside `.explore/targets/<case-id>/`, or a check refers to a `[T<n>]`
|
|
58
58
|
that has no card. A blocking question (`🚫 TODO [blocking]`) is the normal way to wait for a person's answer.
|
|
59
59
|
|
|
60
|
+
**`pnpm case:check` fails on a marker** — the same claim carries two signals: a marker word inside a question block, a
|
|
61
|
+
marker on the line a question hangs under, or a marker inside the `## Values` yaml (where it would become part of the
|
|
62
|
+
expected value). Keep one of the two, and move a `## Values` marker to the prose line above the block. A marker without
|
|
63
|
+
a reason is only a warning: write `🔍 to be confirmed: <what a person must decide>`.
|
|
64
|
+
|
|
60
65
|
**`pnpm spec:lint` fails** — read the rule name in the message: `no-sleep` (`waitForTimeout`), `no-networkidle`,
|
|
61
66
|
download handling, a screenshot or image check without `// target: T<n>`, a PDF check on a magic number alone. The
|
|
62
67
|
fixes are in the framework's `generating-specs` conventions; do not silence the rule.
|
|
@@ -117,6 +122,41 @@ cases, specs, `profiles.yaml` and `package.json` stay. Then run `pnpm dlx @asrun
|
|
|
117
122
|
`claude mcp list`; `failed: ... path specified` means the `playwright` path in that file does not resolve (repair it
|
|
118
123
|
with `pnpm asrun init --force <file>`).
|
|
119
124
|
|
|
125
|
+
**A `browser_*` tool says `Tool "<name>" not found`** — expected for part of them: the `playwright-test` MCP
|
|
126
|
+
(`run-test-mcp-server`) lists every browser tool Playwright has, but a call runs in a paused test whose backend installs
|
|
127
|
+
only the `core` and `testing` capabilities, plus it never installs the `skillOnly` tools. The server is up and the other
|
|
128
|
+
tools work: use the substitute in `.claude/skills/exploring-cases/references/tools.md` and do not report a missing MCP
|
|
129
|
+
server. `browser_highlight` / `browser_hide_highlight` become `locator.highlight()` / `page.hideHighlight()` through
|
|
130
|
+
`browser_run_code_unsafe`, `browser_network_clear` becomes a marker on the request list,
|
|
131
|
+
`browser_start_recording` / `browser_stop_recording` are never called (the handoff card,
|
|
132
|
+
`.claude/skills/exploring-cases/references/handoff-card.js`, starts and stops the recording itself and returns
|
|
133
|
+
`recorded`). `browser_annotate` cannot run either: the person names the element in the handoff card. `pnpm asrun doctor`
|
|
134
|
+
prints how many tools the installed Playwright runs and warns when `tools.md` names one it cannot.
|
|
135
|
+
|
|
136
|
+
**An AI run stops answering and never finishes** — the log keeps repeating `... still waiting for <tool call> (running
|
|
137
|
+
5m 30s)` or `... no output from the agent for 5m`, and nothing else arrives. A workspace has exactly one
|
|
138
|
+
`playwright-test` MCP server (the `run-test-mcp-server` child process OpenCode starts from `opencode.jsonc`), shared by
|
|
139
|
+
every session in that workspace, and Playwright runs one paused test in it at a time. One interrupted browser call —
|
|
140
|
+
closing the browser window by hand, cancelling a run while a call is in flight, or a second AI run using the same
|
|
141
|
+
server — leaves that queue wedged: the call never returns and every later run hangs on its first
|
|
142
|
+
`generator_setup_page` until the process is replaced. `pnpm asrun doctor` cannot see it, because it starts a fresh
|
|
143
|
+
server of its own. Cancel the stuck run, quit and restart OpenCode (that ends its MCP processes), then start the run
|
|
144
|
+
again. From the UI this is bounded: only one AI run at a time runs per workspace (the `ai` job lock, whatever case it is
|
|
145
|
+
on), and a run that says nothing for 10 minutes — 15 while a handoff card waits for a person — is stopped with this
|
|
146
|
+
instruction in its log.
|
|
147
|
+
|
|
148
|
+
**The UI server restarted while an AI run was going** — the job is still in the Jobs drawer, marked **Interrupted**,
|
|
149
|
+
and its log stops where the server died. If the agent (and the `playwright-test` MCP server below it) is still running,
|
|
150
|
+
the job offers **Stop the leftover process**; stop it before starting another AI run, or both agents share one MCP
|
|
151
|
+
server and wedge it. A job that had already written `RESULT: needs-input` is recovered as **Succeeded**, so its answers
|
|
152
|
+
can still be sent. Every finished job has **Run again**, which reuses the inputs it was started with.
|
|
153
|
+
|
|
154
|
+
**A call fails with `Missing key`** — the published schema requires two things that are easy to leave out: `intent` on
|
|
155
|
+
every action, input and assertion tool (`browser_click`, `browser_type`, `browser_evaluate`, `browser_run_code_unsafe`,
|
|
156
|
+
`browser_wait_for`, ...), and the fields that have a default (`browser_take_screenshot` needs `scale: "css"`,
|
|
157
|
+
`browser_network_requests` needs `static: false`, `generator_setup_page` needs `plan`). The full list is in
|
|
158
|
+
`.claude/skills/exploring-cases/references/tools.md`.
|
|
159
|
+
|
|
120
160
|
**An AI job runs but uses a slow or limited model** — the top bar's **Model** picker sets the model for UI runs;
|
|
121
161
|
without one, a one-shot OpenCode run uses its built-in free model, which has a daily usage limit. Set `"model"` in
|
|
122
162
|
`opencode.jsonc` to stop depending on the browser's choice. For Claude Code, the picker lists what its settings,
|
package/docs/web-ui.md
CHANGED
|
@@ -62,14 +62,15 @@ Single, between **Panel**, **Editor** and **Doc**.
|
|
|
62
62
|
- **Source and preview** scroll together, the preview block at the caret is shaded, and a click in the preview puts
|
|
63
63
|
the caret on that line. `Ctrl/Cmd+S` saves.
|
|
64
64
|
- **What needs a person is highlighted** in both: `TODO` in any spelling (red for `🚫 TODO [blocking]`, gold
|
|
65
|
-
otherwise), `🔍 to be confirmed
|
|
65
|
+
otherwise), `🔍 to be confirmed` / `👁️ not observed` / `TBC` / `TBD` in blue, and your `✍️ Answer:` lines in
|
|
66
66
|
green. Hover for a tooltip; the small X after a marker removes it (or the whole question block for a `TODO`) in one
|
|
67
67
|
step, with Undo; the file is saved when you save it.
|
|
68
68
|
- **New case** takes an optional folder and runs `pnpm case:new`.
|
|
69
69
|
- **The Panel** shows the case as a form, with the open questions first (accept one to delete the question, or write an
|
|
70
70
|
answer and the AI folds it in). Then the front matter (title, login, suites, writes and its note, environments), the
|
|
71
|
-
steps (sentence, `[writes]` / `[input]` / `[email]` tags, `- Ready when:` lines,
|
|
72
|
-
`## Expected` and `## Visual checks` bullets (drag to reorder), the `## Values` table (one
|
|
71
|
+
`## Rules` bullets, the steps (sentence, `[writes]` / `[input]` / `[email]` tags, `- Ready when:` lines,
|
|
72
|
+
`- Recipe:` lines), the `## Expected` and `## Visual checks` bullets (drag to reorder), the `## Values` table (one
|
|
73
|
+
column per profile; an
|
|
73
74
|
empty cell inherits `default`), and the `## Targets` cards read-only, with what each is missing. With two or more
|
|
74
75
|
profiles the Values section offers a **Rows: Keys / Profiles** switch, and from four profiles on it also adds a
|
|
75
76
|
search and environment chips. With more profiles than keys it starts as one row per profile, so every name and
|
|
@@ -78,6 +79,11 @@ Single, between **Panel**, **Editor** and **Doc**.
|
|
|
78
79
|
the card: what it is, where, how it is identified, what it must not be, and its reference screenshots. Every section
|
|
79
80
|
collapses and has a **?** with a short help card. Panel edits land in the same draft as the source editor, so one
|
|
80
81
|
save covers both, and **Discard** throws them away together.
|
|
82
|
+
- **The To confirm section** lists every line the AI marked: the sentence, where it is (`## Values · line 42`), **why it
|
|
83
|
+
needs you** (the reason the AI wrote after the marker's colon), **what it feeds** (an assertion or only a note) and,
|
|
84
|
+
when it saved one, the **screenshot** from the exploration, so you look at the image instead of opening the page
|
|
85
|
+
again. Rows that change an assertion are highlighted; **Confirm** settles one line and **Confirm all** settles the
|
|
86
|
+
list in one undo step. None of this blocks generating a spec; healing stops until the case is settled.
|
|
81
87
|
- **Spec view** is editable too (`Ctrl/Cmd+S`; refused when the file changed on disk meanwhile). **Open in editor**
|
|
82
88
|
opens it in the default application for `.spec.ts`, **Copy path** copies its absolute path.
|
|
83
89
|
- Click the file name (or the pencil icon) to **rename** or move the case: the spec and the baselines move with it and
|
|
@@ -121,6 +127,21 @@ One entry per run, with a PRODUCTION or STAGING badge (runs from before environm
|
|
|
121
127
|
The bottom bar shows every command started from the UI with live stdout/stderr and the exit code; a job can be
|
|
122
128
|
cancelled there. Opening it shrinks the page instead of covering it.
|
|
123
129
|
|
|
130
|
+
- Jobs are recorded under `.tmp/ui-jobs/<id>/` (`record.json`, `chunks.jsonl`, `output.log`), kept for 7 days and the
|
|
131
|
+
newest 50. A UI server restart does not forget them. A job that was running is shown as **Interrupted**: its output
|
|
132
|
+
stops at the last line the server received (the UI replays the last 4 MB of structured output; `output.log` keeps the
|
|
133
|
+
whole text). If its process is still alive, the page offers **Stop the leftover process** (the agent, its browser and
|
|
134
|
+
its MCP server), and a new AI run or test run is refused until it is stopped: one agent per workspace, as above. A
|
|
135
|
+
job that had already written `RESULT: needs-input` is recovered as **Succeeded**, so its answers can still be sent.
|
|
136
|
+
- **Run again** starts a finished job with the inputs it was started with. A heal keeps its hints, model, profiles and
|
|
137
|
+
report, so nothing has to be typed again. It is a new job with a new conversation and report; write cases, baseline
|
|
138
|
+
overwrites and an existing spec are confirmed again.
|
|
139
|
+
|
|
140
|
+
- Only one AI job runs at a time per workspace: `/explore-case`, `/gen-spec` and `/heal-spec` all drive the workspace's
|
|
141
|
+
single `playwright-test` MCP server, and two at once wedge it. Starting a second one is refused with the name of the
|
|
142
|
+
job it clashes with. A run that says nothing for 10 minutes — 15 while a handoff card waits for a person — is stopped
|
|
143
|
+
and its log says why; see [troubleshooting](./troubleshooting.md#ai-steps).
|
|
144
|
+
|
|
124
145
|
- An AI job's log is structured: a timestamp, one line per tool call (name, what it touched, how long it took), the
|
|
125
146
|
first lines of what it returned with the colors the command printed, the agent's thinking dimmed while **Thinking**
|
|
126
147
|
is ticked, and its messages rendered as Markdown. Long output is a preview with a count. The same text is kept in
|
|
@@ -128,8 +149,10 @@ cancelled there. Opening it shrinks the page instead of covering it.
|
|
|
128
149
|
- When an AI job ends, a thin **What to do next** bar appears under the log (or **Details** opens it in a sheet):
|
|
129
150
|
open questions with an answer box and **Use the suggested defaults**, the confirmations the exploration marked, and
|
|
130
151
|
the next action (**Generate spec**, **Open Run tests**, **View spec**, **Re-explore**). A run that ends with
|
|
131
|
-
`needs-input` gets an amber *Needs your input* badge; sending answers continues the same conversation, once per turn
|
|
132
|
-
|
|
152
|
+
`needs-input` gets an amber *Needs your input* badge; sending answers continues the same conversation, once per turn.
|
|
153
|
+
The record survives a UI server restart, so a run that had already written its questions can still be answered and a
|
|
154
|
+
finished job can be started again with **Run again**; after a restart, a job that did not get that far is answered in
|
|
155
|
+
the case file and run again.
|
|
133
156
|
- A run that ends without its `RESULT:` line gets an amber *No result* badge and a banner under the command instead of
|
|
134
157
|
a "what to do next" bar: nothing is known about what it did, so read the log before trusting the case or the spec.
|
|
135
158
|
A run that is not allowed to write files (plan mode, a read-only sandbox) ends that way; the framework starts
|
|
@@ -154,7 +177,7 @@ cancelled there. Opening it shrinks the page instead of covering it.
|
|
|
154
177
|
|
|
155
178
|
| Path | What |
|
|
156
179
|
|---|---|
|
|
157
|
-
| `.tmp/ui-jobs/` | the log of every job started from the UI (git-ignored) |
|
|
180
|
+
| `.tmp/ui-jobs/` | the record, chunks and log of every job started from the UI (git-ignored); 7 days, newest 50 |
|
|
158
181
|
| `.tmp/hints/` | the text you type in the Heal dialog, passed to the agent as `--hints` |
|
|
159
182
|
| `.explore/` | reference screenshots of the target cards (git-ignored) |
|
|
160
183
|
| `reports/` | one folder per run, `reports/latest` points at the newest |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@asrunjs/asrun",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.0",
|
|
4
4
|
"description": "Multi-profile end-to-end test suite: plain-language cases, AI-written Playwright specs, evidence for every run.",
|
|
5
5
|
"license": "UNLICENSED",
|
|
6
6
|
"author": "Zoro Chen",
|
|
@@ -55,6 +55,7 @@
|
|
|
55
55
|
"cleanup": "tsx scripts/cleanup.ts",
|
|
56
56
|
"case:new": "tsx scripts/new-case.ts",
|
|
57
57
|
"case:check": "tsx scripts/check-cases.ts",
|
|
58
|
+
"markers:confirm": "tsx scripts/markers-confirm.ts",
|
|
58
59
|
"spec:lint": "tsx scripts/lint-specs.ts",
|
|
59
60
|
"baselines:review": "tsx scripts/review-baselines.ts",
|
|
60
61
|
"pdf:render": "tsx scripts/render-pdf.ts",
|
|
@@ -6,9 +6,9 @@ Load the framework skill `{{framework}}/.claude/skills/exploring-cases/SKILL.md`
|
|
|
6
6
|
|
|
7
7
|
Rules combine in this order, and a later one wins where they conflict: the framework baseline
|
|
8
8
|
`{{framework}}/AGENTS.md`, the framework skill, `.claude/skills/exploring-cases.md` if it exists, then the workspace
|
|
9
|
-
`AGENTS.md` §2 (Project rules). F1–F6 always win. Project facts are in `AGENTS.md` §3.
|
|
9
|
+
`AGENTS.md` §2 (Project rules). F1–F6 and R2 always win. Project facts are in `AGENTS.md` §3.
|
|
10
10
|
|
|
11
|
-
Arguments: `<case-id> <env> [--attended] [--profiles a,b]`. The case id is the path below `cases/` without `.md` (it may include a
|
|
11
|
+
Arguments: `<case-id> <env> [--attended] [--profiles a,b] [--hints <file>]`. The case id is the path below `cases/` without `.md` (it may include a
|
|
12
12
|
group folder, e.g. `documents/landing-page`) and is required. `<env>` is `staging` or `production`; it is required and there
|
|
13
13
|
is no default (without one, stop and say so). With `--attended` I am at the machine and the browser window is
|
|
14
14
|
visible: when a step writes data or needs a choice, hand it to me as the skill describes, wait for me, and record
|
|
@@ -16,12 +16,17 @@ what I did as a general recipe under the step it belongs to (`- Recipe:` lines).
|
|
|
16
16
|
`--profiles a,b` explores only those profiles (comma-separated ids from `profiles.yaml`); without it, at least one
|
|
17
17
|
logged-in profile per domain. A profile the list leaves out is not opened and nothing is written about it: runs add up,
|
|
18
18
|
so a later run covers the rest.
|
|
19
|
+
`--hints <file>` (or free text after the arguments) is what I know or want checked; read it first. After the work, a
|
|
20
|
+
reusable instruction goes into `## Rules` by the gate in
|
|
21
|
+
`{{framework}}/.claude/skills/exploring-cases/references/case-rules.md`; a one-off clue stays in the report.
|
|
19
22
|
|
|
20
|
-
|
|
23
|
+
Read `## Rules` before the steps and apply it. Follow
|
|
24
|
+
`{{framework}}/.claude/skills/exploring-cases/references/case-rules.md` (a rule says how to run the case, never what it
|
|
25
|
+
proves; never delete or rewrite one), `assertion-targets.md` (list the candidates, never
|
|
21
26
|
pick when in doubt, ask for the exact image/text/place/UI) and `readiness.md` (wait for the key requests and the
|
|
22
27
|
content before you read anything) of the skill.
|
|
23
28
|
|
|
24
29
|
This command edits `cases/<case-id>.md` only (plus screenshots under `.explore/targets/`). Do not ask me questions in
|
|
25
|
-
the chat: write open questions into the case file as the skill describes, and end with the `RESULT:` line
|
|
26
|
-
not allowed to write files at all (a plan-mode run,
|
|
27
|
-
`RESULT: needs-input`.
|
|
30
|
+
the chat: write open questions into the case file as the skill describes, and end with the `RESULT:` line as plain text
|
|
31
|
+
on the last line, never in a code fence or in emphasis. If you are not allowed to write files at all (a plan-mode run,
|
|
32
|
+
a read-only sandbox), say what blocked you and end with `RESULT: needs-input`.
|
|
@@ -7,18 +7,23 @@ $ARGUMENTS
|
|
|
7
7
|
|
|
8
8
|
Rules combine in this order, and a later one wins where they conflict: the framework baseline
|
|
9
9
|
`{{framework}}/AGENTS.md`, the framework skill, `.claude/skills/generating-specs.md` if it exists, then the workspace
|
|
10
|
-
`AGENTS.md` §2 (Project rules). F1–F6 always win. Project facts are in `AGENTS.md` §3.
|
|
10
|
+
`AGENTS.md` §2 (Project rules). F1–F6 and R2 always win. Project facts are in `AGENTS.md` §3.
|
|
11
11
|
|
|
12
|
-
Arguments: `<case-id> [<env>] [--force]`. The case file is `cases/<case-id>.md` (the id may include one group folder,
|
|
12
|
+
Arguments: `<case-id> [<env>] [--force] [--hints <file>]`. The case file is `cases/<case-id>.md` (the id may include one group folder,
|
|
13
13
|
e.g. `documents/landing-page`). `<env>` is the environment to explore in (`staging` or `production`); there is no default. If
|
|
14
14
|
the spec already exists, it is left alone unless `--force` is given; with `--force` it is regenerated and overwritten.
|
|
15
|
+
`--hints <file>` (or free text after the arguments) is what I know or want changed; read it first. A reusable
|
|
16
|
+
instruction goes into `## Rules` by the gate in
|
|
17
|
+
`{{framework}}/.claude/skills/exploring-cases/references/case-rules.md`; a one-off clue stays in the report.
|
|
15
18
|
There is no profile option: the spec is generated for every profile the case holds information for (its `## Values`
|
|
16
19
|
keys, its target cards and its observations), not for a chosen subset and not only for the profiles that are logged in.
|
|
17
20
|
|
|
18
|
-
Follow the skill's procedure exactly, including
|
|
21
|
+
Follow the skill's procedure exactly, including `case-rules.md` (read `## Rules` before the steps; a rule says how to
|
|
22
|
+
run the case, never what it proves), its `assertion-targets.md` and `readiness.md`; run
|
|
19
23
|
`pnpm spec:lint <case-id>`. Only generate the spec: do not run `pnpm asrun` (running specs is a separate
|
|
20
24
|
process), and do not create baselines.
|
|
21
25
|
|
|
22
26
|
Do not ask me questions: resolve ambiguity as the skill describes (explore the page, use observed values marked for
|
|
23
|
-
QA, write the remaining questions into the case file), and end with the `RESULT:` line
|
|
24
|
-
files at all (a plan-mode run, a read-only
|
|
27
|
+
QA, write the remaining questions into the case file), and end with the `RESULT:` line as plain text on the last line,
|
|
28
|
+
never in a code fence or in emphasis. If you are not allowed to write files at all (a plan-mode run, a read-only
|
|
29
|
+
sandbox), say what blocked you and end with `RESULT: needs-input`.
|
|
@@ -7,7 +7,7 @@ spec", and follow it exactly.
|
|
|
7
7
|
|
|
8
8
|
Rules combine in this order, and a later one wins where they conflict: the framework baseline
|
|
9
9
|
`{{framework}}/AGENTS.md`, the framework skill, `.claude/skills/generating-specs.md` if it exists, then the workspace
|
|
10
|
-
`AGENTS.md` §2 (Project rules). F1–F6 always win. Project facts are in `AGENTS.md` §3.
|
|
10
|
+
`AGENTS.md` §2 (Project rules). F1–F6 and R2 always win. Project facts are in `AGENTS.md` §3.
|
|
11
11
|
|
|
12
12
|
Arguments: $ARGUMENTS
|
|
13
13
|
Format: `<case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]`, with optional free text after them.
|
|
@@ -16,23 +16,27 @@ Format: `<case-id> <env> [--profiles a,b] [--report <run-id>] [--hints <file>]`,
|
|
|
16
16
|
- `--profiles`: the profiles to run when evidence is needed. Without it, every profile that is logged in in `<env>`.
|
|
17
17
|
- `--report`: the report (folder name under `reports/`) to read first. Without it, the newest report of this case.
|
|
18
18
|
- `--hints <file>`: a file with what the user wants repaired or improved, with any evidence. Read it first. Any free
|
|
19
|
-
text after the arguments is the same kind of request. Both are optional.
|
|
19
|
+
text after the arguments is the same kind of request. Both are optional. Read `## Rules` before the steps and apply
|
|
20
|
+
it; after the work, a reusable method instruction goes into `## Rules` by the gate in
|
|
21
|
+
`{{framework}}/.claude/skills/exploring-cases/references/case-rules.md`, while a one-off clue stays in the report.
|
|
20
22
|
|
|
21
23
|
A passing run is not a reason to refuse a request: healing also refines a case and its spec. Never lower the bar for a
|
|
22
24
|
product finding to make a test pass.
|
|
23
25
|
|
|
24
26
|
Do not ask me questions in the chat: this may run unattended. Resolve what the report, the case and the page can
|
|
25
|
-
resolve, and write what still needs a person into the case as a `[blocking]` question. End with the `RESULT:` line
|
|
26
|
-
|
|
27
|
-
write files at all (a plan-mode run, a read-only
|
|
28
|
-
and end with `RESULT: needs-input`: never answer
|
|
27
|
+
resolve, and write what still needs a person into the case as a `[blocking]` question. End with the `RESULT:` line as
|
|
28
|
+
plain text on the last line, never in a code fence or in emphasis: `ready` when the healing or the refinement is
|
|
29
|
+
finished, `needs-input` when something is still open. If you could not write files at all (a plan-mode run, a read-only
|
|
30
|
+
sandbox, a refused permission), say in one sentence what blocked you and end with `RESULT: needs-input`: never answer
|
|
31
|
+
with a plan and a question instead.
|
|
29
32
|
|
|
30
33
|
Before you decide "product or test", audit the assertion itself (target card, baseline content, readiness): see step 5
|
|
31
34
|
of the skill's healing section, `assertion-targets.md` and `readiness.md`. Run `pnpm spec:lint <case-id>`.
|
|
32
35
|
Read the existing report first (`pnpm heal:evidence --case <case-id> --env <env>`), and run the case again only when
|
|
33
36
|
the skill says it is needed. Always run with `--logged-in`. If the change leaves an open question or an unconfirmed
|
|
34
37
|
marker in the case, stop there, tell me to review the case and run `/explore-case`, and end with
|
|
35
|
-
`RESULT: needs-input`.
|
|
38
|
+
`RESULT: needs-input`. A marker inside `## Rules` does not stop the spec unless the rule covers a literal the spec
|
|
39
|
+
asserts. Otherwise tell me whether it is a test problem or a product finding.
|
|
36
40
|
|
|
37
41
|
Run `pnpm asrun ...`, `pnpm heal:evidence ...`, `pnpm spec:lint ...`, `pnpm baselines:review ...` and
|
|
38
42
|
`pnpm status ...` as plain commands, optionally piped to `tail` or `head`, and do not chain other commands with them
|
|
@@ -7,7 +7,7 @@ documents: $ARGUMENTS
|
|
|
7
7
|
|
|
8
8
|
Rules combine in this order, and a later one wins where they conflict: the framework baseline
|
|
9
9
|
`{{framework}}/AGENTS.md`, the framework skill, `.claude/skills/making-cases.md` if it exists, then the workspace
|
|
10
|
-
`AGENTS.md` §2 (Project rules). F1–F6 always win. Project facts are in `AGENTS.md` §3.
|
|
10
|
+
`AGENTS.md` §2 (Project rules). F1–F6 and R2 always win. Project facts are in `AGENTS.md` §3.
|
|
11
11
|
|
|
12
12
|
The input is either text or a path to a file with the notes. Follow the skill's procedure exactly. Ask me the open
|
|
13
13
|
questions before writing any case file. Do not generate specs; that is `/gen-spec`.
|
package/templates/AGENTS.md
CHANGED
|
@@ -9,11 +9,13 @@ Read this file and the framework baseline it points at, before any AI step.
|
|
|
9
9
|
|
|
10
10
|
Rules are applied in this order, and a later layer wins where they conflict (the framework baseline, "Precedence"):
|
|
11
11
|
|
|
12
|
-
1. `F1`–`F6` of the framework baseline: safety, in every project. Nothing here can change them.
|
|
12
|
+
1. `F1`–`F6` and `R2` of the framework baseline: safety, in every project. Nothing here can change them.
|
|
13
13
|
2. The framework baseline, `{{framework}}/AGENTS.md`.
|
|
14
14
|
3. The framework skill (`{{framework}}/.claude/skills/<skill>/SKILL.md`).
|
|
15
15
|
4. A project skill override, `.claude/skills/<skill>.md` (optional; the command and the skill read it).
|
|
16
16
|
5. This file, §2 below: what this project does differently.
|
|
17
|
+
6. A case's `## Rules`: how to run that one case, never what is true. It is below §2 and cannot weaken `F1`–`F6` or
|
|
18
|
+
`R2`.
|
|
17
19
|
|
|
18
20
|
The framework skills are loaded by the commands in `.claude/commands/` (Claude Code) and `.opencode/commands/`
|
|
19
21
|
(OpenCode). The skills live in the framework package; §3 holds this project's facts.
|
|
@@ -21,8 +23,8 @@ The framework skills are loaded by the commands in `.claude/commands/` (Claude C
|
|
|
21
23
|
## 2. Project rules
|
|
22
24
|
|
|
23
25
|
Only what differs from the baseline: name the rule id, and whether it `narrows` (stricter) or `replaces` (a different
|
|
24
|
-
default, for example the artifact language) the baseline rule. `replaces` is refused for `F1`–`F6`. Rows
|
|
25
|
-
cell is not a rule id are ignored, so the placeholder below can stay until you need it.
|
|
26
|
+
default, for example the artifact language) the baseline rule. `replaces` is refused for `F1`–`F6` and `R2`. Rows
|
|
27
|
+
whose first cell is not a rule id are ignored, so the placeholder below can stay until you need it.
|
|
26
28
|
|
|
27
29
|
| Rule | Project rule | Effect |
|
|
28
30
|
|---|---|---|
|
|
@@ -39,9 +41,10 @@ Examples (delete what you do not use; the fenced block is an example, not a rule
|
|
|
39
41
|
`pnpm asrun doctor` fails when this table names a `[non-negotiable]` rule, an unknown rule id, or an `Effect` that is
|
|
40
42
|
not `narrows` or `replaces`.
|
|
41
43
|
|
|
42
|
-
The structural tokens are fixed English whatever the project language: the section headings (`## Values`, `##
|
|
43
|
-
...), a step's `- Recipe (phase): ...` lines, `### T<n>`, the `TODO` blocks with their `Answer:` line,
|
|
44
|
-
`[input]`/`[writes]`/`[email]`, `RESULT
|
|
44
|
+
The structural tokens are fixed English whatever the project language: the section headings (`## Values`, `## Rules`,
|
|
45
|
+
`## Targets`, ...), a step's `- Recipe (phase): ...` lines, `### T<n>`, the `TODO` blocks with their `Answer:` line,
|
|
46
|
+
`[input]`/`[writes]`/`[email]`, `RESULT:` (plain text on the last line, never inside a code fence or in emphasis),
|
|
47
|
+
and the front-matter keys. See "Fixed tokens" in the framework baseline.
|
|
45
48
|
|
|
46
49
|
## 3. Project facts
|
|
47
50
|
|
|
@@ -11,6 +11,13 @@ suites: [smoke, regression] # a case may be in several suites; filter with `pn
|
|
|
11
11
|
## Goal
|
|
12
12
|
Why this case exists, in one or two sentences.
|
|
13
13
|
|
|
14
|
+
## Rules
|
|
15
|
+
Optional instructions for the AI on how to run this case (a page quirk, a mandatory read-only preparation, a control
|
|
16
|
+
to avoid). They never state what the case expects and never allow writing data or acting for a person. One bullet per
|
|
17
|
+
rule; a bullet may start with `Step 3:`, `Before step 3:` or `After step 3:`. Delete this paragraph when the case needs
|
|
18
|
+
none.
|
|
19
|
+
- <e.g. Before step 1: dismiss the cookie banner; it covers the header.>
|
|
20
|
+
|
|
14
21
|
## Steps
|
|
15
22
|
1. Open <page or path, e.g. /account/>.
|
|
16
23
|
- Ready when: <the one positive signal that what this step loads is really there, e.g. the list shows its first
|
package/templates/package.json
CHANGED
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
"cleanup": "asrun cleanup",
|
|
18
18
|
"case:new": "asrun case:new",
|
|
19
19
|
"case:check": "asrun case:check",
|
|
20
|
+
"markers:confirm": "asrun markers:confirm",
|
|
20
21
|
"spec:lint": "asrun spec:lint",
|
|
21
22
|
"baselines:review": "asrun baselines:review",
|
|
22
23
|
"pdf:render": "asrun pdf:render",
|