@asrunjs/asrun 0.8.0 → 0.10.0

package/.claude/skills/exploring-cases/references/open-questions.md

@@ -5,7 +5,7 @@ without anybody present (a one-shot `claude -p` / `opencode run`). The case file
 talk to each other, so nothing is lost between runs, tools or sessions.
 
 ## How a question is written in the case file
-Every marker starts with its emoji, which is part of the marker: `🚫 TODO [blocking]`, `❓ TODO [non-blocking]`, `✍️ Answer:`, `🔍 to be confirmed by QA` and `👁️ not observed`. The web UI colors the emoji with the marker and offers an X button that deletes it.
+Every marker starts with its emoji, which is part of the marker: `🚫 TODO [blocking]`, `❓ TODO [non-blocking]`, `✍️ Answer:`, `🔍 to be confirmed` and `👁️ not observed`. The web UI colors the emoji with the marker and offers an X button that deletes it. `(seen on <profile>)` is not a marker: it is a plain note, shown as ordinary text.
 Every question is a small block that hangs under a normal sentence. The sentence is the agent's suggestion, written
 exactly as it should read in the final case. The block holds the question and an empty answer slot:
 
@@ -15,7 +15,7 @@ exactly as it should read in the final case. The block holds the question and an
     - ✍️ Answer:
 ```
 
-The same shape is used under a step, under a bullet of "Expected" and under the yaml block of "Values":
+The same shape is used under a step, under a bullet of "Expected" and on the prose line above the `## Values` block:
 
 ```
 11. Delete the document with the "Delete" option of its "..." menu on the list page. [writes]
@@ -40,8 +40,52 @@ Rules for writing it:
   QA should confirm it later but nothing waits for it.
 - One question per block; narrow broad questions down instead of copying them ("which profile elements?" becomes
   "is the logo the only per-profile element in the left side bar?").
-- A value read from a page is not a question: write it as a statement marked `(observed, 🔍 to be confirmed by QA)`.
-  It needs no TODO block; the reviewer confirms it by deleting the marker.
+- A fact read from a page is not a question: either say it as a plain note `(seen on <profile>)`, or mark it
+  `(observed on <profile>, 🔍 to be confirmed: <what a person must decide>)` when a wrong reading would still pass.
+  It needs no TODO block. See "Marker or question, never both" below.
+
+## Where the block hangs
+The parent line is the sentence the question is about: the panel shows it on its own, next to "The text above this
+question", so it has to make sense without the rest of the document.
+- **The immediate line, not an ancestor.** Hang the block under the one sentence that makes the claim. Under a
+  nested bullet, indent it under that bullet, not under the top-level one above it; when the claim only makes sense
+  with an umbrella sentence, write the claim itself as a top-level bullet and hang the block there.
+- **One claim, one block.** Several bullets raising one question get one block, under the closest of them, with the
+  others' evidence in that block. Never end a long list with a single question that covers all of it.
+- **A sentence that stands alone.** No "see above", no pronoun whose referent is in a sibling bullet.
+- **Never inside the `## Values` yaml block.** A marker or a question there becomes part of the expected value: put
+  it on the prose line above the block.
+
+## Marker or question, never both
+A sentence carries at most one "not settled yet" signal, and the two marks are alternatives:
+- `🚫/❓ TODO`: there **is** a question. The parent sentence is a suggestion, the block says what to decide.
+- `(observed on <profile>, 🔍 to be confirmed: <what a person must decide>)`: there is **no** question, only a check
+  somebody has to make later. The part after the colon is required: "to be confirmed by QA" alone tells a person
+  nothing, and the panel shows that part as "why it needs you".
+- `(observed on <profile>, evidence: .explore/observations/<case-id>/<image>.png)`: the same, with a screenshot the
+  reviewer opens instead of going back to the page. Save it while exploring.
+- `(seen on <profile>)`: a plain note. The run checks it, so nobody has to; it gets no colour and no panel row.
+
+- Never put a marker inside a TODO block: not in the question, not in the evidence, not in a wrapped line. The words
+  "to be confirmed", "TBC", "TBD" and "not observed" are marker words even in prose there, and `pnpm case:check` fails
+  on them.
+- Never put a marker on the sentence a block hangs under: keep the question and drop the marker.
+- `👁️ not observed` is for an expectation that came from the notes and was never seen on a page. It is never the
+  evidence of a question.
+
+## Which sentences carry a marker
+A marker is for exactly one thing: a fact a person must settle because a wrong reading would still pass. Ask yourself:
+**if this line were read wrong, would the generated spec fail?**
+- **It would fail** → no marker. Locators, the flow of a step, navigation, request URLs, readiness signals, whether an
+  element exists. The run is the confirmation. Leave `(seen on <profile>)` when where it was seen matters.
+- **It would pass anyway** → marker, with the reason. The literals a check asserts (a `## Values` entry, the wording of
+  an `## Expected` bullet) and the identity of a `[T<n>]` target. This is the class that hides a wrong expectation
+  forever; `assertion-targets.md` records two real incidents of it.
+- A `## Rules` bullet the agent inferred from a page carries a marker only when the rule changes what is asserted;
+  otherwise it is `(seen on <profile>)`. A rule a person stated never carries one.
+
+After a passing run, `pnpm markers:confirm --case <id> --env <env>` settles the markers whose line the spec checks
+(`// unconfirmed: <the sentence>`), so a person does not have to open the page again to see whether it is still true.
 
 ## How the user answers
 Two ways, and the agent must understand both on the next run:
@@ -69,7 +113,9 @@ answer, adds `(page shows <x> on <profile-id>)` and reports it as a finding. It
 ## Resolution order (no question goes to the user while a better step exists)
 1. **The page answers it.** Facts such as exact labels, which controls exist, where the logo is, what a menu holds
    are answered by exploration (`/explore-case`; needs an environment and a logged-in profile; the agent only reads, a person does the steps that write). Rewrite the parent sentence
-   with the fact, mark it `(observed, 🔍 to be confirmed by QA)` and remove the block.
+   with the fact and remove the block. Then decide the mark by "Which sentences carry a marker": a fact the run checks
+   gets `(seen on <profile>)`, a fact that would pass while wrong gets
+   `(observed on <profile>, 🔍 to be confirmed: <what a person must decide>)`.
    - A check with no expected value ("check X", "can see Y") becomes: the observed behaviour, marked as observed.
      A spec may be generated from it.
    - Values that a check asserts (logo name, label, option list) are observed per profile and written into the case's
@@ -101,16 +147,22 @@ answer, adds `(page shows <x> on <profile-id>)` and reports it as a finding. It
 
 ## Final report
 Report in the user's language. List every open block with its question and the parent sentence it would confirm, so
-the user can see what deleting it accepts. The last line of the final message is exactly one of:
+the user can see what deleting it accepts. List the markers separately, each with its reason and whether it changes an
+assertion: that is the short list a person actually has to look at. The last line of the final message is plain text
+and nothing else, exactly one of:
 
 ```
 RESULT: ready
 RESULT: needs-input
 ```
 
+Never put that line in a code fence or inline code, never make it bold, a heading or a list item, and put no label,
+number or emoji in front of it: the panel reads it to show how the run ended. (The block above only lists the two
+tokens; write the one that applies as the last line of your message, not the block.)
+
 - `ready`: no `[blocking]` question remains. Non-blocking ones may remain; list them so QA can confirm them.
 - `needs-input`: at least one `[blocking]` question remains in the case file (list each with its parent sentence),
-  or the work could not be done (no environment, nobody logged in). Say what the user should edit in the case file,
-  then re-run.
+  or the work could not be done (no environment, nobody logged in, or no permission to write files: a plan-mode run,
+  a read-only sandbox). Say what the user should edit in the case file, then re-run.
 
 Do not write anything after the `RESULT:` line.

package/.claude/skills/exploring-cases/references/readiness.md

@@ -22,9 +22,10 @@ A check for something **missing, empty or hidden** is only valid after a positiv
 means: the list finished loading and has rows, and none is X.
 
 ## Exploring
-- Know the key requests: after the action, call `browser_network_requests`, pick the data calls (XHR/fetch to the
-  app's API, the ones that return the content). Wait until they are finished; use `browser_network_clear` before the
-  action so the list only holds what the action started.
+- Know the key requests: after the action, call `browser_network_requests` (`static: false`), pick the data calls
+  (XHR/fetch to the app's API, the ones that return the content). Wait until they are finished. `browser_network_clear`
+  is not runnable in this MCP, so take a marker instead: read the list before the action, keep the highest number, and
+  after the action only the entries above it belong to it (`tools.md`, "Network requests without a clear").
 - Then wait for the content signal (`browser_wait_for` with text, or `browser_find` that the content is there), and
   take the snapshot only then. A snapshot taken right after a click is a frame, not a fact.
 - Observations record the ready signal: `(observed after <signal>, ...)`. A fact with no signal is not recorded.

package/.claude/skills/exploring-cases/references/tools.md

@@ -0,0 +1,117 @@
+# The browser tools this MCP can run, and the ones it only lists
+
+Shared by `exploring-cases`, `generating-specs` and healing. It is binding in all three: read it before the first
+browser call of a run.
+
+`playwright run-test-mcp-server` answers `tools/list` with **every** browser tool Playwright has, but a call runs
+inside a paused test, and that backend installs one capability set:
+
+```js
+const config = { capabilities: ["testing"] };   // playwright/lib/mcp/test/browserBackend.js
+const toolList = tools.filteredTools(config);   // core* + testing, minus `skillOnly`
+```
+
+Everything outside that set is listed and still cannot run: it answers `Tool "<name>" not found`. Two groups:
+- a capability other than `core`/`testing` (`devtools`, `storage`, `network`, `pdf`, `vision`, `config`);
+- `skillOnly: true` tools, which the backend never installs (`browser_network_clear`, `browser_console_clear`).
+
+**`Tool "<name>" not found` is not a broken MCP server.** The server is up and every other tool works. Never stop
+with `RESULT: needs-input` for it, never report a missing MCP server, and never fall back to an ad-hoc Playwright
+script (it sees a different browser and its network view is not `browser_network_requests`, see A2). Use the
+substitute below instead, and say in the report which substitute you used.
+
+`pnpm asrun doctor` prints how many tools the installed Playwright actually runs and warns when this file names one
+it cannot (it happens when Playwright changes the filter; the file then needs the same edit).
+
+## Runnable
+
+Navigation and reading:
+`browser_navigate`, `browser_navigate_back`, `browser_snapshot`, `browser_find`, `browser_evaluate`,
+`browser_run_code_unsafe`, `browser_tabs`, `browser_resize`, `browser_close`, `browser_handle_dialog`,
+`browser_console_messages`, `browser_emulate_media`, `browser_wait_for`.
+
+Interaction: `browser_click`, `browser_type`, `browser_fill_form`, `browser_press_key`, `browser_hover`,
+`browser_drag`, `browser_drop`, `browser_select_option`, `browser_file_upload`.
+
+Screenshots: `browser_take_screenshot`.
+
+Network: `browser_network_requests`, `browser_network_request`.
+
+Test helpers: `browser_generate_locator`, `browser_verify_element_visible`, `browser_verify_text_visible`,
+`browser_verify_list_visible`, `browser_verify_value`.
+
+`browser_run_code_unsafe` takes Playwright's own API (a `page`, `context`, `expect` are in scope), which is what
+every substitute below uses.
+
+## Listed but not runnable
+
+| Tool | Why | Do this instead |
+|---|---|---|
+| `browser_highlight`, `browser_hide_highlight` | `devtools` | [Outline a target](#outlining-a-target-without-browser_highlight) |
+| `browser_network_clear` | `skillOnly` | [Network requests without a clear](#network-requests-without-a-clear) |
+| `browser_console_clear` | `skillOnly` | read `browser_console_messages` and note the messages you already saw |
+| `browser_start_recording`, `browser_stop_recording` | `devtools` | the handoff card (`user-handoff.md`, "Recording") |
+| `browser_annotate` | `devtools` | let the person name the element in the handoff card, then outline it yourself |
+| `browser_start/stop_tracing`, `browser_start/stop_video`, `browser_video_*`, `browser_resume` | `devtools` | not needed by a case |
+| `browser_cookie_*`, `browser_localstorage_*`, `browser_sessionstorage_*`, `browser_storage_state`, `browser_set_storage_state` | `storage` | never: saved logins are passwords (F5) |
+| `browser_route`, `browser_unroute`, `browser_route_list`, `browser_network_state_set` | `network` | never: a case checks the real page |
+| `browser_pdf_save` | `pdf` | a page that serves a PDF is downloaded and rendered (`pnpm pdf:render`) |
+| `browser_mouse_move_xy`, `browser_mouse_down`, `browser_mouse_up`, `browser_mouse_wheel`, `browser_mouse_click_xy`, `browser_mouse_drag_xy` | `vision` | `browser_click` with a locator |
+| `browser_get_config` | `config` | `pnpm status --env <env>` |
+
+## Arguments the schema requires
+
+A call that looks complete still fails when one of these is missing. Both come from the MCP's own schema, and both
+answer `Missing key`:
+
+1. **`intent` on every action, input and assertion tool.** The test runner adds it so the recorded step has a name.
+   One short sentence, imperative, about what this call is for: `intent: "open the article preview"`. It belongs on
+   `browser_navigate`, `browser_navigate_back`, `browser_click`, `browser_type`, `browser_fill_form`,
+   `browser_press_key`, `browser_hover`, `browser_drag`, `browser_drop`, `browser_select_option`,
+   `browser_file_upload`, `browser_evaluate`, `browser_run_code_unsafe`, `browser_tabs`, `browser_close`,
+   `browser_resize`, `browser_handle_dialog`, `browser_emulate_media`, and on `browser_wait_for` and every
+   `browser_verify_*`. The read-only tools (`browser_snapshot`, `browser_find`, `browser_take_screenshot`,
+   `browser_console_messages`, `browser_network_requests`, `browser_network_request`, `browser_generate_locator`) take
+   none.
+2. **A field with a default is published as required:**
+   - `browser_take_screenshot` → `scale: "css"` (CSS pixels, the size baselines expect).
+   - `browser_network_requests` → `static: false` (data calls only).
+   - `browser_network_request` → `index` (from the numbered list).
+   - `browser_tabs` → `action` (`"list"`, `"new"`, `"select"` or `"close"`).
+   - `browser_console_messages` → `level` (`"error"` is enough).
+   - `browser_handle_dialog` → `accept` (`true` to accept, `false` to dismiss).
+   - `generator_setup_page` → `project`, `seedFile` and `plan` (`plan: ""` when there is no plan).
+
+## Outlining a target without `browser_highlight`
+
+`browser_highlight` is a one-line wrapper over `locator.highlight()`, and that method is public Playwright API. The
+overlay is the real one, not an approximation, and nothing of the page is changed.
+
+1. Outline (one call, one element, the card's `Identified by` as the locator):
+   ```js
+   // browser_run_code_unsafe({intent: 'outline the target for the reference screenshot', code: ...})
+   async (page) => {
+     const target = page.getByRole('img', {name: '<the Identified by value of the card>'});
+     await target.highlight({style: 'outline: 3px solid #e11d48; outline-offset: 2px'});
+     return await target.toString();
+   }
+   ```
+2. Screenshot the page (not the element) so the reader sees where it sits:
+   `browser_take_screenshot({scale: 'css', filename: '.explore/targets/<case-id>/T<n>-<profile-id>.png'})`
+3. Remove it before the next step: `browser_run_code_unsafe` with `intent: 'remove the highlight overlay'` and
+   `async (page) => { await page.hideHighlight(); }`.
+
+A highlight overlay is not page state: it never appears in a snapshot, it is not part of what the case checks, and
+`page.hideHighlight()` clears all of them. Do this for every candidate when several elements are compared, and say in
+the card what the screenshot shows.
+
+## Network requests without a clear
+
+`browser_network_requests` returns a numbered list that only grows while the page stays open, so a marker works:
+
+1. Before the action: `browser_network_requests({static: false})`, note the highest number in the list (`mark`).
+2. Do the action.
+3. Read `browser_network_requests({static: false})` again and take the entries above `mark`: those, and only those,
+   were started by the action. `browser_network_request({index})` gives one request's detail and status.
+
+Record the marker as a plain note in the report (`after 12 new requests; the data call was 7`), never in the case.

package/.claude/skills/exploring-cases/references/user-handoff.md

@@ -29,7 +29,9 @@ that nobody watches.
    copied with one click.
 3. Never ask the user to send email (production: never; other environments: only when `email` is `true` for the
    profile). Never ask for login or 2FA. Name everything the user creates `[asrun] <something>`.
-4. Start recording with `browser_start_recording` (see "Recording").
+4. Do not start a recording yourself: the card starts and stops it (see "Recording"). Never call
+   `browser_start_recording` or `browser_stop_recording`: the `playwright-test` MCP lists them but cannot run them
+   (`devtools` tools; see `tools.md`).
 
 ## The wait loop
 Run it with one `browser_run_code_unsafe` call. A single call may wait for minutes (verified: 150 s without a
@@ -63,8 +65,9 @@ inside the call that runs the card: OpenCode reports a tool call only when it en
 way the web UI can see that a person is needed while the card waits. With the card alone there is no "Your turn"
 badge, no count in the tab title and no notification (a HINT in the job log says so).
 
-1. Set the options on the browser context (this call only sets data):
+1. Set the options on the browser context (this call only sets data; `browser_run_code_unsafe` needs `intent`):
    ```js
+   // browser_run_code_unsafe({intent: 'set the handoff card options', code: ...})
    async (page) => {
      page.context().__handoffCfg = {
        message: '<what the user does, one sentence>',
@@ -77,11 +80,16 @@ badge, no count in the tab title and no notification (a HINT in the job log says
    ```
 2. Run the card and wait: `browser_run_code_unsafe` with `filename` set to the absolute path of
    `<framework>/.claude/skills/exploring-cases/references/handoff-card.js` (use the absolute path of the repository, a
-   relative one is resolved against the workspace root, which may be the parent folder). It returns `{outcome: 'done' | 'skip' | 'condition' | 'timeout', waitedMs}`.
+   relative one is resolved against the workspace root, which may be the parent folder) and an `intent` such as
+   `'hand off step 5 to the person'`. It returns
+   `{outcome: 'done' | 'skip' | 'condition' | 'timeout', waitedMs, recording, recorded}`: `recorded` holds the
+   actions the person performed, one Playwright line each, across every tab (`page`, `page1`, ...), and is empty when
+   the recording was unavailable (`recording: false`; `recordingError` says why).
 
 Check once, after the card shows, that it looks right: take a screenshot and see a title bar with a move icon, a
-chevron button at its right and, when `copy` is not empty, a line per text with a copy icon. If not, stop and report
-it; do not hand off with a broken card.
+chevron button at its right and, when `copy` is not empty, a line per text with a copy icon. Playwright's own recording
+toolbar at the top of the page is expected and is not part of the card. If the card is not right, stop and report it;
+do not hand off with a broken card.
 
 Tell the user in the chat (the terminal or the UI job log) as well: "Your turn in the browser window: <task>. Press
 Done on the card (bottom-left; drag it if it is in the way) when you finish." When the run was started from the web
@@ -105,17 +113,21 @@ The Done button is always there (also on the minimized card) as the fallback, an
   for two seconds before it is removed, so the user knows when they may let go.
 - `outcome: 'timeout'` or `'skip'`: do not guess. Write a `TODO [blocking]` under the step and go on with what does
   not depend on it.
-- Take a snapshot and write down what the user did (see "Recording"), then continue read-only.
+- Take a snapshot, write down what the user did from the card's `recorded` (see "Recording"), then continue read-only.
 - At the end of the exploration hand off once more when the user created data: "Delete the [asrun] items you
   created, then press Done". The case must clean up after itself, and so must the exploration.
 
 ## Recording
-Between the start and the end of a handoff the agent records what the user does (`browser_start_recording`,
-`browser_stop_recording`). The recorded Playwright code is **raw material, never saved as it is**. It is turned into
-a general recipe and written under the step as `- Recipe (phase): ...` lines (format below). The recipe holds what the
-user did in words, plus the **labels and roles of the controls** they used ("the textbox 'Title'", "the button
-'Next'"), because `/gen-spec` cannot write data and so cannot see those controls itself. It never holds selectors,
-ids or the names of data that can change.
+The card records what the person does: it starts the recording when it shows and stops it on every way out (Done,
+"Cannot do this", the finish condition, the timeout, an error), in the same `browser_run_code_unsafe` call. The agent
+neither starts nor stops it. The result is the card's `recorded`: one Playwright line per action, `page` for the first
+tab that acted and `page1`, `page2`, ... for the others. The aliases follow the order in which actions first appear,
+not the order in which the tabs were opened, so read the actions before you describe tabs.
+
+The recorded Playwright code is **raw material, never saved as it is**. It is turned into a general recipe and written
+under the step as `- Recipe (phase): ...` lines (format below). The recipe holds what the user did in words, plus the
+**labels and roles of the controls** they used ("the textbox 'Title'", "the button 'Next'"), because `/gen-spec` cannot
+write data and so cannot see those controls itself. It never holds selectors, ids or the names of data that can change.
 
 Make the recipe as general as possible, because lists and data change:
 - **A choice from a list that can change** (items, search results, a contact list): record the position, never the
@@ -123,14 +135,31 @@ Make the recipe as general as possible, because lists and data change:
 - **Random is allowed only when the user asks for it** ("pick any 3"). The default is the position, because it can be
   repeated. A random pick must be written into the test's annotations when the spec runs.
 - **Text the case creates**: `[asrun] <something>`.
-- **A value that has to exist** (an existing name, a search term): write what it must be like ("a name that
-  returns a result") and the observed value as `(observed "<value>", 🔍 to be confirmed by QA)` plus a
-  `[non-blocking]` TODO block. Do not put a brittle literal in as the only truth.
+- **A value that has to exist** (an existing name, a search term): the recipe says what it must be like ("a name
+  that returns a result") and notes the observed value as plain text `(seen on <profile>: "<value>")`. Do not put a
+  brittle literal in as the only truth. One claim carries one signal: when a `[non-blocking]` TODO under the recipe
+  asks whether any such value is acceptable, the observed value gets **no** marker (`open-questions.md`, "Marker or
+  question, never both").
 - **Nothing that depends on the pick** may become an assertion (a total changes with the items). Say so in a
   TODO when a check would.
 - Settings that the user changed on purpose (a toggle, a template) are recorded as they were set, by label.
 
-If the recording tool fails or returns nothing, describe what the user did from before/after snapshots instead.
+The recording is raw: merge and prune it before you write the recipe.
+- **The same action can appear twice**, and clicking a checkbox or its label can give both a `click` and a `check`.
+  Write it once ("check 'Agree'").
+- **Side products**: clicking a textbox to focus it, `Cmd+C`, `Scroll`, `Escape`. Keep only actions that change data
+  or the page's state.
+- **The card's own buttons are filtered out** of `recorded`. A click on the card's text (not a button) can still show
+  up; ignore it.
+- **Not recorded at all**: drag and drop, native file pickers, OS dialogs. Describe those from before/after snapshots.
+
+If `recording` is `false` or `recorded` is empty, do not silently write a recipe from snapshots: write a
+`🚫 TODO [blocking]` under the step that says the step was not recorded (and why, from `recordingError` when there is
+one). A recipe from before/after snapshots may be added next to that block only, and must say that it came from
+snapshots, not from a recording.
+
+If Playwright's own recording toolbar shows at the top of the page while the card waits, it is not AsRun's: leave it
+alone.
 
 ## Recipe lines under a step
 One or more `- Recipe (phase): ...` lines directly under the `[input]` or `[writes]` step they belong to, indented by
@@ -139,7 +168,7 @@ keep in sync.
 
 ```
 3. Select an existing document and pick 3 items. [input]
-    - Recipe (start): search for an existing document and pick result #1 (observed search term "...", 🔍 to be confirmed by QA).
+    - Recipe (start): search for an existing document and pick result #1 (seen on example-staging: "quarterly report").
       - ❓ TODO [non-blocking]: Is any result that opens fine as the starting point? (seen on example-staging)
         - ✍️ Answer:
     - Recipe (items): select 3 items by position (#1 to #3) from the list.