@tamsensedev/runner 0.13.0 → 0.18.0

package/library/skills/ui-testing/SKILL.md

@@ -1,102 +1,102 @@
----
-name: ui-testing
-description: Use when a change affects what users see or click in a web app and should be checked in a real browser, when writing or fixing Playwright or Cypress tests, or when a test step must verify UI behaviour. Runs the project's E2E suite or a headless check from a temp folder and reports screenshots and errors as evidence.
----
-
-# UI testing in a headless browser
-
-Reading the code tells you what should happen; a browser shows what does. This skill gets the app running, drives the changed flows headless, and turns what it saw into evidence for the report. Nobody watches the run, so every step must finish on its own and leave the machine as it was.
-
-The project's own setup wins: its E2E framework, config, fixtures, page objects, test data, auth helpers and scripts. Use them before anything from here.
-
-## Hard rules
-
-1. **Headless only.** Never open a window: no `--headed`, `--ui`, `--debug`, `show-report`, `show-trace`, `cypress open`, `PWDEBUG`. Playwright's HTML reporter opens a browser on failure when not in CI, so run with `--reporter=line` and `PW_TEST_HTML_REPORT_OPEN=never`. Set `BROWSER=none` when starting dev servers so they do not open one either.
-2. **No new dependencies.** Do not add packages to the repository and do not download browsers (`playwright install`, `npx some-tool@latest`) just to check. Installing the project's own dependencies from its lockfile (`pnpm install --frozen-lockfile`, `npm ci`) is fine: a fresh checkout often has none. If the project has no Playwright but the `TAMSENSE_PLAYWRIGHT` environment variable is set, the TAMsense runner has already installed `playwright-core` and a headless browser outside the project, and the headless check script uses them. If nothing is, check from the code and say the browser check could not run.
-3. **A local copy from the task folder, with the project's own start command, on a free port.** Start the app with its dev or preview script, never a hand-made server that skips its config. Find its local setup yourself, use only local settings from this checkout, check that every address points at this computer, and run its containers as your own compose project: see [references/local-stand.md](references/local-stand.md). Prefer the headless check script, which starts the server and stops its whole process tree for you. Stop it at the end, including child processes, and confirm the port is free again.
-8. **Stop only what you started, by process id.** This is someone's own computer and their other programs may be running. Never kill processes by name or pattern: no `taskkill /IM`, `pkill`, `killall`, `Stop-Process -Name`. If you start a server yourself, record its PID and stop that process tree (`taskkill /PID <pid> /T /F` on Windows, `kill -- -<pgid>` elsewhere). If a process you started cannot be stopped that way, say so in the report and leave the rest alone.
-4. **Outputs outside the repository.** Screenshots, traces, reports, scratch scripts and auth state go to a new folder in the OS temp directory. `git status` must show nothing new from you when you finish.
-5. **Never fake it.** If the app cannot start (missing environment variables, secrets, a database or service you may not use), report exactly what is missing. Do not mock the whole backend and call the result a pass, and do not invent screenshots or results.
-6. **Do not write to shared systems.** If the settings point at a shared, staging or production backend or database, do not start the app: the check could not run (local-stand.md, rule 7). Recorded browsers also abort requests that change data on other computers.
-7. **Never hide failures.** Do not skip, `fixme`, delete or loosen a failing test to get a green run. In a check step, a failing test is a finding.
-9. **Every browser is recorded for the task.** In a TAMsense run each Playwright context becomes a replay people watch on the task, and the tested site's TAMsense tracker is blocked. Open contexts through the recorder with a title and a purpose, mark the meaningful steps, close each with notes on what you tested and what happened, and add a video for canvas, maps, video players or iframes from another site: see [references/recording.md](references/recording.md).
-
-## Process
-
-### 1. Decide what to check
-
-From the task, the spec's acceptance criteria and the diff (`git diff origin/<base>...HEAD --stat`, then the changed components, routes and styles), write a short list of scenarios before touching the browser:
-
-- The main flow the task changed, end to end.
-- Its states: loading, empty, error, long or unusual content, many items, disabled controls.
-- A narrow screen (390 px wide) if layout or styles changed, plus a desktop width.
-- Keyboard use for any new or changed control (Tab to it, activate with Enter or Space, Escape closes overlays). The `accessibility` skill has the full checks.
-
-### 2. Find how this project runs and tests its UI
-
-- E2E setup: `playwright.config.*`, `cypress.config.*`, `e2e/` or `tests/` folders, scripts like `test:e2e` in `package.json`, and the CI workflow steps that run them (they show the exact command, env vars and services).
-- The app: dev and preview scripts, required env vars (`.env.example`, config validation, README), services it needs (database, cache, other apps in a monorepo). Check whether required variables are set by name; never print their values.
-- Auth: seeded test users, a test login endpoint, existing storage-state setup.
-- The installed Playwright or Cypress version.
-
-### 3a. The project has an E2E suite
-
-Run the specs related to the change, then the whole suite if it is fast enough.
-
-- Playwright: `PW_TEST_HTML_REPORT_OPEN=never npx playwright test <specs> --reporter=line --output <tmp>/test-results` (use the package manager the project uses). Its browsers are recorded by themselves under the step's name; a long suite fills the run's limit of 20 replays, so run the related specs rather than everything when you can. Its `webServer` config usually starts the app: check the port and `reuseExistingServer`, and make sure no stale server of yours is still running.
-- Cypress: `npx cypress run --spec <specs> --config screenshotsFolder=<tmp>/screens,videosFolder=<tmp>/videos,video=false` (headless by default).
-- If a run fails, rerun the failing spec alone to separate a real failure from a flaky one, and read the error and screenshot. Decide whether the failure comes from the change (the diff touches that flow) or existed before; say which and why.
-
-In a code step, add or update tests for the changed flow following the suite's structure; see [references/playwright-patterns.md](references/playwright-patterns.md) for locators, waiting, auth, mocking and the plan, write, repair loop. In a review or test step, do not edit the repository's tests; check extra scenarios with 3b instead.
-
-### 3b. No suite, or scenarios the suite does not cover
-
-Use the headless check in [references/headless-check.md](references/headless-check.md): a Node.js script you write to a temp folder that resolves Playwright from the project, starts the app with the project's command on a free port, visits the pages at 1280 and 390 px, runs your steps with marks, records each page for the task, saves screenshots and a JSON summary (console errors, failed requests, horizontal overflow, optional axe scan), and stops the app.
-
-If the project has no Playwright among its dependencies and no other headless browser tool is available, do not add one. Check the change from the code and say so.
-
-### 4. Look at the evidence
-
-- Open every screenshot with the Read tool and compare it with what the scenario expects. Describe what you see in plain words: what is on the screen, what is wrong, where.
-- Console errors, page errors and failed requests from the changed code are findings; third-party noise is not, but name it so the reader knows you saw it.
-- Sideways scrolling at 390 px, cut-off text, overlapping elements, missing states and dead buttons are findings.
-- For visual quality and consistency with the design system, apply the `design-review` skill; for the rules list, `web-interface-guidelines`; for accessibility, `accessibility`.
-
-### 5. Clean up
-
-Stop the servers you started, by their process id only (hard rule 8), check the port no longer answers, remove containers and temp folders you created unless the task asks to keep artifacts, and run `git status` in every repository to confirm you left nothing behind.
-
-## Report
-
-```
-UI check: <passed | failed | could not run>
-How: <suite command or headless check>, browser <chromium | chrome | msedge>, widths <1280, 390>
-Recordings: <titles of the replays and videos attached to the task>
-Scenarios:
-- <scenario>: pass | fail | not checked (why). Evidence: <what the screenshot or log shows>
-Findings:
-- <file:line if traced> <what is wrong, how to reproduce, what was expected>
-Not checked: <and why: missing env vars X, Y; no browser available; flow needs an external provider>
-Assumptions: <test account used, data you created locally, version>
-```
-
-In a test step, base the verdict on the scenarios: a broken acceptance criterion, an error from the changed code or a layout broken on narrow screens fails it. If the local copy could not start because of this computer (Docker not running, no local setup, settings that point outside), finish as [references/local-stand.md](references/local-stand.md) rule 7 says: `couldNotRun`, no handoff, the task waits for a person. If the browser check could not run for another reason, do not report it as passed; judge from the other evidence (tests, code) and state clearly that the UI was not seen. If the change itself stops the app from starting or building, that is a failure.
-
-## Common mistakes
-
-- Assuming the page works because the build passed.
-- Fixed sleeps instead of waiting for the element that proves readiness.
-- Letting the HTML reporter or a dev server open a browser window.
-- Leaving the dev server running, or `test-results/` and screenshots inside the repository.
-- Testing only the happy path at desktop width.
-- Pointing the check at staging or production because the local app did not start.
-- Listing screenshot paths without saying what they show.
-- Generic names for screenshots and recordings ("screenshot1", "test run") instead of what they show and why.
-
-## Done when
-
-- Every scenario from step 1 is checked or listed as not checked with a reason.
-- Evidence (described screenshots, error lines, failing assertions) backs each finding.
-- No window was opened, no dependency added, nothing left running or written into the repository.
-
-Sources: the start, wait, act, stop approach of anthropics/skills webapp-testing (Apache-2.0, commit dbd4588), testdino-hq/playwright-skill (MIT, commit 178d1cb), ideas from the Playwright test agents (Apache-2.0). Changed: rewritten for Node and the project's own setup, unattended rules added.
+---
+name: ui-testing
+description: Use when a change affects what users see or click in a web app and should be checked in a real browser, when writing or fixing Playwright or Cypress tests, or when a test step must verify UI behaviour. Runs the project's E2E suite or a headless check from a temp folder and reports screenshots and errors as evidence.
+---
+
+# UI testing in a headless browser
+
+Reading the code tells you what should happen; a browser shows what does. This skill gets the app running, drives the changed flows headless, and turns what it saw into evidence for the report. Nobody watches the run, so every step must finish on its own and leave the machine as it was.
+
+The project's own setup wins: its E2E framework, config, fixtures, page objects, test data, auth helpers and scripts. Use them before anything from here.
+
+## Hard rules
+
+1. **Headless only.** Never open a window: no `--headed`, `--ui`, `--debug`, `show-report`, `show-trace`, `cypress open`, `PWDEBUG`. Playwright's HTML reporter opens a browser on failure when not in CI, so run with `--reporter=line` and `PW_TEST_HTML_REPORT_OPEN=never`. Set `BROWSER=none` when starting dev servers so they do not open one either.
+2. **No new dependencies.** Do not add packages to the repository and do not download browsers (`playwright install`, `npx some-tool@latest`) just to check. Installing the project's own dependencies from its lockfile (`pnpm install --frozen-lockfile`, `npm ci`) is fine: a fresh checkout often has none. If the project has no Playwright but the `TAMSENSE_PLAYWRIGHT` environment variable is set, the TAMsense runner has already installed `playwright-core` and a headless browser outside the project, and the headless check script uses them. If nothing is, check from the code and say the browser check could not run.
+3. **A local copy from the task folder, with the project's own start command, on a free port.** Start the app with its dev or preview script, never a hand-made server that skips its config. Find its local setup yourself, use only local settings from this checkout, check that every address points at this computer, and run its containers as your own compose project: see [references/local-stand.md](references/local-stand.md). Prefer the headless check script, which starts the server and stops its whole process tree for you. Stop it at the end, including child processes, and confirm the port is free again.
+8. **Stop only what you started, by process id.** This is someone's own computer and their other programs may be running. Never kill processes by name or pattern: no `taskkill /IM`, `pkill`, `killall`, `Stop-Process -Name`. If you start a server yourself, record its PID and stop that process tree (`taskkill /PID <pid> /T /F` on Windows, `kill -- -<pgid>` elsewhere). If a process you started cannot be stopped that way, say so in the report and leave the rest alone.
+4. **Outputs outside the repository.** Screenshots, traces, reports, scratch scripts and auth state go to a new folder in the OS temp directory. `git status` must show nothing new from you when you finish.
+5. **Never fake it.** If the app cannot start (missing environment variables, secrets, a database or service you may not use), report exactly what is missing. Do not mock the whole backend and call the result a pass, and do not invent screenshots or results.
+6. **Do not write to shared systems.** If the settings point at a shared, staging or production backend or database, do not start the app: the check could not run (local-stand.md, rule 7). Recorded browsers also abort requests that change data on other computers.
+7. **Never hide failures.** Do not skip, `fixme`, delete or loosen a failing test to get a green run. In a check step, a failing test is a finding.
+9. **Every browser is recorded for the task.** In a TAMsense run each Playwright context becomes a replay people watch on the task, and the tested site's TAMsense tracker is blocked. Open contexts through the recorder with a title and a purpose, mark the meaningful steps, close each with notes on what you tested and what happened, and add a video for canvas, maps, video players or iframes from another site: see [references/recording.md](references/recording.md).
+
+## Process
+
+### 1. Decide what to check
+
+From the task, the spec's acceptance criteria and the diff (`git diff origin/<base>...HEAD --stat`, then the changed components, routes and styles), write a short list of scenarios before touching the browser:
+
+- The main flow the task changed, end to end.
+- Its states: loading, empty, error, long or unusual content, many items, disabled controls.
+- A narrow screen (390 px wide) if layout or styles changed, plus a desktop width.
+- Keyboard use for any new or changed control (Tab to it, activate with Enter or Space, Escape closes overlays). The `accessibility` skill has the full checks.
+
+### 2. Find how this project runs and tests its UI
+
+- E2E setup: `playwright.config.*`, `cypress.config.*`, `e2e/` or `tests/` folders, scripts like `test:e2e` in `package.json`, and the CI workflow steps that run them (they show the exact command, env vars and services).
+- The app: dev and preview scripts, required env vars (`.env.example`, config validation, README), services it needs (database, cache, other apps in a monorepo). Check whether required variables are set by name; never print their values.
+- Auth: seeded test users, a test login endpoint, existing storage-state setup.
+- The installed Playwright or Cypress version.
+
+### 3a. The project has an E2E suite
+
+Run the specs related to the change, then the whole suite if it is fast enough.
+
+- Playwright: `PW_TEST_HTML_REPORT_OPEN=never npx playwright test <specs> --reporter=line --output <tmp>/test-results` (use the package manager the project uses). Its browsers are recorded by themselves under the step's name; a long suite fills the run's limit of 20 replays, so run the related specs rather than everything when you can. Its `webServer` config usually starts the app: check the port and `reuseExistingServer`, and make sure no stale server of yours is still running.
+- Cypress: `npx cypress run --spec <specs> --config screenshotsFolder=<tmp>/screens,videosFolder=<tmp>/videos,video=false` (headless by default).
+- If a run fails, rerun the failing spec alone to separate a real failure from a flaky one, and read the error and screenshot. Decide whether the failure comes from the change (the diff touches that flow) or existed before; say which and why.
+
+In a code step, add or update tests for the changed flow following the suite's structure; see [references/playwright-patterns.md](references/playwright-patterns.md) for locators, waiting, auth, mocking and the plan, write, repair loop. In a review or test step, do not edit the repository's tests; check extra scenarios with 3b instead.
+
+### 3b. No suite, or scenarios the suite does not cover
+
+Use the headless check in [references/headless-check.md](references/headless-check.md): a Node.js script you write to a temp folder that resolves Playwright from the project, starts the app with the project's command on a free port, visits the pages at 1280 and 390 px, runs your steps with marks, records each page for the task, saves screenshots and a JSON summary (console errors, failed requests, horizontal overflow, optional axe scan), and stops the app.
+
+If the project has no Playwright among its dependencies and no other headless browser tool is available, do not add one. Check the change from the code and say so.
+
+### 4. Look at the evidence
+
+- Open every screenshot with the Read tool and compare it with what the scenario expects. Describe what you see in plain words: what is on the screen, what is wrong, where.
+- Console errors, page errors and failed requests from the changed code are findings; third-party noise is not, but name it so the reader knows you saw it.
+- Sideways scrolling at 390 px, cut-off text, overlapping elements, missing states and dead buttons are findings.
+- For visual quality and consistency with the design system, apply the `design-review` skill; for the rules list, `web-interface-guidelines`; for accessibility, `accessibility`.
+
+### 5. Clean up
+
+Stop the servers you started, by their process id only (hard rule 8), check the port no longer answers, remove containers and temp folders you created unless the task asks to keep artifacts, and run `git status` in every repository to confirm you left nothing behind.
+
+## Report
+
+```
+UI check: <passed | failed | could not run>
+How: <suite command or headless check>, browser <chromium | chrome | msedge>, widths <1280, 390>
+Recordings: <titles of the replays and videos attached to the task>
+Scenarios:
+- <scenario>: pass | fail | not checked (why). Evidence: <what the screenshot or log shows>
+Findings:
+- <file:line if traced> <what is wrong, how to reproduce, what was expected>
+Not checked: <and why: missing env vars X, Y; no browser available; flow needs an external provider>
+Assumptions: <test account used, data you created locally, version>
+```
+
+In a test step, base the verdict on the scenarios: a broken acceptance criterion, an error from the changed code or a layout broken on narrow screens fails it. If the local copy could not start because of this computer (Docker not running, no local setup, settings that point outside), finish as [references/local-stand.md](references/local-stand.md) rule 7 says: `couldNotRun`, no handoff, the task waits for a person. If the browser check could not run for another reason, do not report it as passed; judge from the other evidence (tests, code) and state clearly that the UI was not seen. If the change itself stops the app from starting or building, that is a failure.
+
+## Common mistakes
+
+- Assuming the page works because the build passed.
+- Fixed sleeps instead of waiting for the element that proves readiness.
+- Letting the HTML reporter or a dev server open a browser window.
+- Leaving the dev server running, or `test-results/` and screenshots inside the repository.
+- Testing only the happy path at desktop width.
+- Pointing the check at staging or production because the local app did not start.
+- Listing screenshot paths without saying what they show.
+- Generic names for screenshots and recordings ("screenshot1", "test run") instead of what they show and why.
+
+## Done when
+
+- Every scenario from step 1 is checked or listed as not checked with a reason.
+- Evidence (described screenshots, error lines, failing assertions) backs each finding.
+- No window was opened, no dependency added, nothing left running or written into the repository.
+
+Sources: the start, wait, act, stop approach of anthropics/skills webapp-testing (Apache-2.0, commit dbd4588), testdino-hq/playwright-skill (MIT, commit 178d1cb), ideas from the Playwright test agents (Apache-2.0). Changed: rewritten for Node and the project's own setup, unattended rules added.