@tamsensedev/runner 0.13.0 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +5 -5
- package/README.md +76 -57
- package/dist/cli.js +757 -379
- package/dist/cli.js.map +1 -1
- package/dist/recorder.cjs.map +1 -1
- package/library/.claude-plugin/plugin.json +5 -5
- package/library/THIRD_PARTY_NOTICES.md +123 -123
- package/library/agents/code-architect.md +67 -67
- package/library/agents/code-explorer.md +61 -61
- package/library/agents/code-reviewer.md +73 -73
- package/library/agents/code-simplifier.md +62 -62
- package/library/agents/design-critic.md +52 -52
- package/library/agents/second-opinion.md +64 -64
- package/library/agents/security-auditor.md +86 -86
- package/library/agents/silent-failure-hunter.md +70 -70
- package/library/agents/test-coverage-analyzer.md +78 -78
- package/library/agents/test-runner.md +48 -48
- package/library/agents/ui-tester.md +50 -50
- package/library/licenses/Apache-2.0.txt +202 -202
- package/library/skills/accessibility/SKILL.md +144 -144
- package/library/skills/accessibility/references/wcag-checks.md +168 -168
- package/library/skills/addressing-review-feedback/SKILL.md +109 -109
- package/library/skills/app-screens/SKILL.md +96 -96
- package/library/skills/app-screens/references/shell-kpis-charts.md +108 -108
- package/library/skills/app-screens/references/states-density-phone.md +54 -54
- package/library/skills/app-screens/references/tables-forms-settings.md +104 -104
- package/library/skills/avoiding-generic-ai-design/SKILL.md +92 -92
- package/library/skills/avoiding-generic-ai-design/references/tells.md +112 -112
- package/library/skills/clean-commits/SKILL.md +85 -85
- package/library/skills/design-review/SKILL.md +107 -107
- package/library/skills/design-review/references/critique-passes.md +101 -101
- package/library/skills/design-review/references/screenshot-procedure.md +121 -121
- package/library/skills/differential-security-review/LICENSE +427 -427
- package/library/skills/differential-security-review/SKILL.md +105 -105
- package/library/skills/differential-security-review/references/adversarial.md +115 -115
- package/library/skills/differential-security-review/references/methodology.md +145 -145
- package/library/skills/differential-security-review/references/patterns.md +108 -108
- package/library/skills/false-positive-check/LICENSE +427 -427
- package/library/skills/false-positive-check/SKILL.md +95 -95
- package/library/skills/false-positive-check/references/bug-class-verification.md +128 -128
- package/library/skills/false-positive-check/references/deep-verification.md +113 -113
- package/library/skills/false-positive-check/references/evidence-templates.md +93 -93
- package/library/skills/false-positive-check/references/false-positive-patterns.md +122 -122
- package/library/skills/false-positive-check/references/gate-reviews.md +36 -36
- package/library/skills/false-positive-check/references/standard-verification.md +80 -80
- package/library/skills/focused-implementation/SKILL.md +104 -104
- package/library/skills/implementation-plan/SKILL.md +66 -66
- package/library/skills/interface-design/SKILL.md +144 -144
- package/library/skills/interface-design/references/craft-defaults.md +130 -130
- package/library/skills/interface-design/references/new-direction.md +62 -62
- package/library/skills/interface-design/references/writing.md +52 -52
- package/library/skills/interface-polish/SKILL.md +103 -103
- package/library/skills/interface-polish/references/detail-rules.md +154 -154
- package/library/skills/interface-polish/references/worst-case-data.md +158 -158
- package/library/skills/motion-design/SKILL.md +106 -106
- package/library/skills/motion-design/references/recipes.md +243 -243
- package/library/skills/postgres-performance/SKILL.md +97 -97
- package/library/skills/postgres-performance/references/rules.md +265 -265
- package/library/skills/project-orientation/SKILL.md +77 -77
- package/library/skills/reviewing-changes/SKILL.md +183 -183
- package/library/skills/safe-database-changes/SKILL.md +111 -111
- package/library/skills/safe-database-changes/references/backfills.md +75 -75
- package/library/skills/safe-database-changes/references/migration-tools.md +69 -69
- package/library/skills/safe-database-changes/references/postgres-ddl.md +199 -199
- package/library/skills/security-review/SKILL.md +132 -132
- package/library/skills/security-review/references/checks-by-area.md +107 -107
- package/library/skills/security-review/references/do-not-report.md +86 -86
- package/library/skills/spec-writing/SKILL.md +144 -144
- package/library/skills/systematic-debugging/SKILL.md +130 -130
- package/library/skills/systematic-debugging/references/flaky-tests.md +51 -51
- package/library/skills/systematic-debugging/references/root-cause-tracing.md +55 -55
- package/library/skills/task-interview/SKILL.md +84 -84
- package/library/skills/test-driven-development/SKILL.md +116 -116
- package/library/skills/test-driven-development/references/writing-good-tests.md +85 -85
- package/library/skills/ui-testing/SKILL.md +102 -102
- package/library/skills/ui-testing/references/headless-check.md +295 -295
- package/library/skills/ui-testing/references/local-stand.md +51 -51
- package/library/skills/ui-testing/references/playwright-patterns.md +134 -134
- package/library/skills/ui-testing/references/recording.md +61 -61
- package/library/skills/verification-before-completion/SKILL.md +113 -113
- package/library/skills/web-interface-guidelines/SKILL.md +202 -202
- package/package.json +46 -46
|
@@ -1,295 +1,295 @@
|
|
|
1
|
-
# Headless check without an E2E suite
|
|
2
|
-
|
|
3
|
-
Read this when the project has no Playwright or Cypress tests you can run for the change, but you still need to see the pages: screenshots, console errors, failed requests, horizontal overflow on narrow screens, and optionally an axe scan.
|
|
4
|
-
|
|
5
|
-
The script below starts the app with the project's own command on a free port, waits for it, opens each page headless at a desktop and a phone width, runs optional steps you write, saves screenshots and a JSON summary to a temp folder, and stops the app and all its child processes at the end. In a TAMsense run each page and width is also recorded as a replay for the task, named from `CONFIG.title` and `CONFIG.purpose` (see [recording.md](recording.md)).
|
|
6
|
-
|
|
7
|
-
## What it needs
|
|
8
|
-
|
|
9
|
-
- Node.js 20 or newer.
|
|
10
|
-
- The `playwright`, `@playwright/test` or `playwright-core` package already installed in the project (it is resolved from the project folder you give it; in a monorepo, use the package folder of the web app). If the project has none and the `TAMSENSE_PLAYWRIGHT` environment variable is set, the script loads the `playwright-core` that the TAMsense runner installed in that folder, with the headless browser from the `TAMSENSE_PLAYWRIGHT_BROWSERS` folder. Nothing is installed.
|
|
11
|
-
- A browser that is already on the machine: Playwright's downloaded Chromium (the project's or the runner's), or the system Chrome or Edge as a fallback. If none works, the script stops and says so. Do not run `playwright install` or add packages to get one; report that the check could not run.
|
|
12
|
-
- `axe-core` is optional: if the project already has it, each page also gets an accessibility scan.
|
|
13
|
-
|
|
14
|
-
## How to use it
|
|
15
|
-
|
|
16
|
-
1. Make a new folder in the OS temp directory (`node -p "require('os').tmpdir()"`), for example `<tmp>/ui-check-<random>`. Never inside the repository.
|
|
17
|
-
2. Write the script there as `ui-check.mjs` and edit only the `CONFIG` block and the `steps` function. Set `title` and `purpose` to what this check is and why ("Feature testing, attempt 2", "Checking the empty orders state after the fix"), and `video: true` when the pages have a canvas, a map, a video player or iframes from another site.
|
|
18
|
-
3. Run `node <tmp>/ui-check-<random>/ui-check.mjs` with a timeout of a few minutes. It prints the summary; screenshots are in `CONFIG.outDir`.
|
|
19
|
-
4. Open the screenshots with the Read tool and look at them. Describe in the report what each one shows; the paths alone mean nothing to the reader.
|
|
20
|
-
5. Delete the folder when the report is written, unless the task asks you to keep artifacts.
|
|
21
|
-
|
|
22
|
-
`startCommand` is the project's own dev or preview command with `{port}` where the port goes, for example `pnpm dev --port {port} --strictPort` (Vite), `npm run dev -- -p {port}` (Next.js), `pnpm nuxt dev --port {port}`, `bin/rails server -p {port}`, `python manage.py runserver 127.0.0.1:{port}`. The script also sets `PORT`. Leave it empty and set `baseUrl` if the app is already running or is static files you serve another way.
|
|
23
|
-
|
|
24
|
-
On Windows the command runs in `cmd.exe`, which may not see the same PATH as your shell. If the summary says the command is not recognized, use absolute paths (for example the full path to `node.exe` and the package manager's script) rather than changing anything in the project.
|
|
25
|
-
|
|
26
|
-
## The script
|
|
27
|
-
|
|
28
|
-
```js
|
|
29
|
-
import { createRequire } from 'node:module'
|
|
30
|
-
import { spawn, execFileSync } from 'node:child_process'
|
|
31
|
-
import { mkdirSync, writeFileSync } from 'node:fs'
|
|
32
|
-
import { join } from 'node:path'
|
|
33
|
-
import net from 'node:net'
|
|
34
|
-
|
|
35
|
-
const CONFIG = {
|
|
36
|
-
projectDir: '/absolute/path/to/web/app',
|
|
37
|
-
outDir: '/absolute/path/to/tmp/ui-check-1234/out',
|
|
38
|
-
startCommand: 'pnpm dev --port {port} --strictPort',
|
|
39
|
-
baseUrl: '',
|
|
40
|
-
paths: ['/'],
|
|
41
|
-
widths: [1280, 390],
|
|
42
|
-
startTimeoutMs: 120_000,
|
|
43
|
-
navigationTimeoutMs: 30_000,
|
|
44
|
-
title: 'Feature testing',
|
|
45
|
-
purpose: 'Checking the changed screens at desktop and phone width',
|
|
46
|
-
video: false,
|
|
47
|
-
}
|
|
48
|
-
|
|
49
|
-
async function steps(page, { path, width, mark }) {
|
|
50
|
-
}
|
|
51
|
-
|
|
52
|
-
const MAX_ITEMS = 20
|
|
53
|
-
const MAX_TITLE = 60
|
|
54
|
-
const serverOutput = []
|
|
55
|
-
const recorder = process.env.TAMSENSE_RECORDER ? createRequire(import.meta.url)(process.env.TAMSENSE_RECORDER) : null
|
|
56
|
-
|
|
57
|
-
function keepTail(chunk) {
|
|
58
|
-
serverOutput.push(...String(chunk).split('\n').filter(Boolean))
|
|
59
|
-
serverOutput.splice(0, Math.max(0, serverOutput.length - 40))
|
|
60
|
-
}
|
|
61
|
-
|
|
62
|
-
function requireFrom(dir, names) {
|
|
63
|
-
const require = createRequire(join(dir, 'package.json'))
|
|
64
|
-
for (const name of names) {
|
|
65
|
-
try {
|
|
66
|
-
return require(name)
|
|
67
|
-
} catch {}
|
|
68
|
-
}
|
|
69
|
-
return null
|
|
70
|
-
}
|
|
71
|
-
|
|
72
|
-
function loadFromProject(names) {
|
|
73
|
-
const found = requireFrom(CONFIG.projectDir, names)
|
|
74
|
-
const runnerCopy = process.env.TAMSENSE_PLAYWRIGHT
|
|
75
|
-
if (found || !names.includes('playwright-core') || !runnerCopy) return found
|
|
76
|
-
if (process.env.TAMSENSE_PLAYWRIGHT_BROWSERS) process.env.PLAYWRIGHT_BROWSERS_PATH = process.env.TAMSENSE_PLAYWRIGHT_BROWSERS
|
|
77
|
-
return requireFrom(runnerCopy, ['playwright-core'])
|
|
78
|
-
}
|
|
79
|
-
|
|
80
|
-
function freePort() {
|
|
81
|
-
return new Promise((resolve, reject) => {
|
|
82
|
-
const server = net.createServer()
|
|
83
|
-
server.on('error', reject)
|
|
84
|
-
server.listen(0, '127.0.0.1', () => {
|
|
85
|
-
const { port } = server.address()
|
|
86
|
-
server.close(() => resolve(port))
|
|
87
|
-
})
|
|
88
|
-
})
|
|
89
|
-
}
|
|
90
|
-
|
|
91
|
-
function startServer(command, port) {
|
|
92
|
-
const child = spawn(command.replaceAll('{port}', String(port)), {
|
|
93
|
-
cwd: CONFIG.projectDir,
|
|
94
|
-
shell: true,
|
|
95
|
-
env: { ...process.env, PORT: String(port), BROWSER: 'none' },
|
|
96
|
-
detached: process.platform !== 'win32',
|
|
97
|
-
windowsHide: true,
|
|
98
|
-
stdio: ['ignore', 'pipe', 'pipe'],
|
|
99
|
-
})
|
|
100
|
-
child.stdout.on('data', keepTail)
|
|
101
|
-
child.stderr.on('data', keepTail)
|
|
102
|
-
return child
|
|
103
|
-
}
|
|
104
|
-
|
|
105
|
-
function stopServer(child) {
|
|
106
|
-
if (!child || child.exitCode !== null) return 'not running'
|
|
107
|
-
try {
|
|
108
|
-
if (process.platform === 'win32') {
|
|
109
|
-
const taskkill = join(process.env.SystemRoot ?? 'C:\\Windows', 'System32', 'taskkill.exe')
|
|
110
|
-
execFileSync(taskkill, ['/pid', String(child.pid), '/T', '/F'], { stdio: 'ignore' })
|
|
111
|
-
} else {
|
|
112
|
-
process.kill(-child.pid, 'SIGKILL')
|
|
113
|
-
}
|
|
114
|
-
return 'stopped'
|
|
115
|
-
} catch (error) {
|
|
116
|
-
child.kill('SIGKILL')
|
|
117
|
-
return `could not stop the process tree of pid ${child.pid}: ${error.message}`
|
|
118
|
-
} finally {
|
|
119
|
-
child.stdout.destroy()
|
|
120
|
-
child.stderr.destroy()
|
|
121
|
-
child.unref()
|
|
122
|
-
}
|
|
123
|
-
}
|
|
124
|
-
|
|
125
|
-
async function waitForServer(url, child) {
|
|
126
|
-
const deadline = Date.now() + CONFIG.startTimeoutMs
|
|
127
|
-
while (Date.now() < deadline) {
|
|
128
|
-
if (child && child.exitCode !== null) throw new Error(`start command exited with code ${child.exitCode}`)
|
|
129
|
-
try {
|
|
130
|
-
await fetch(url, { redirect: 'manual', signal: AbortSignal.timeout(5_000) })
|
|
131
|
-
return
|
|
132
|
-
} catch {}
|
|
133
|
-
await new Promise(resolve => setTimeout(resolve, 1_000))
|
|
134
|
-
}
|
|
135
|
-
throw new Error(`no response from ${url} within ${CONFIG.startTimeoutMs / 1000}s`)
|
|
136
|
-
}
|
|
137
|
-
|
|
138
|
-
async function launchBrowser(chromium) {
|
|
139
|
-
const failures = []
|
|
140
|
-
for (const channel of [undefined, 'chromium', 'chrome', 'msedge']) {
|
|
141
|
-
try {
|
|
142
|
-
return await chromium.launch({ headless: true, channel })
|
|
143
|
-
} catch (error) {
|
|
144
|
-
failures.push(`${channel ?? 'playwright chromium'}: ${String(error.message).split('\n')[0]}`)
|
|
145
|
-
}
|
|
146
|
-
}
|
|
147
|
-
throw new Error(`no browser available without downloading one: ${failures.join(' | ')}`)
|
|
148
|
-
}
|
|
149
|
-
|
|
150
|
-
async function runAxe(page, axeSource) {
|
|
151
|
-
try {
|
|
152
|
-
await page.addScriptTag({ content: axeSource })
|
|
153
|
-
return await page.evaluate(async () => {
|
|
154
|
-
const result = await window.axe.run(document, {
|
|
155
|
-
runOnly: { type: 'tag', values: ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa'] },
|
|
156
|
-
})
|
|
157
|
-
return result.violations.map(violation => ({
|
|
158
|
-
rule: violation.id,
|
|
159
|
-
impact: violation.impact,
|
|
160
|
-
help: violation.help,
|
|
161
|
-
count: violation.nodes.length,
|
|
162
|
-
targets: violation.nodes.slice(0, 5).map(node => JSON.stringify(node.target)),
|
|
163
|
-
}))
|
|
164
|
-
})
|
|
165
|
-
} catch (error) {
|
|
166
|
-
return { error: String(error.message).split('\n')[0] }
|
|
167
|
-
}
|
|
168
|
-
}
|
|
169
|
-
|
|
170
|
-
async function openContext(browser, path, width) {
|
|
171
|
-
const options = { viewport: { width, height: 900 } }
|
|
172
|
-
if (!recorder) return browser.newContext(options)
|
|
173
|
-
const where = `, ${path} at ${width}px`
|
|
174
|
-
const title = `${CONFIG.title.slice(0, MAX_TITLE - where.length)}${where}`
|
|
175
|
-
return recorder.newRecordedContext(browser, { title, purpose: CONFIG.purpose, video: CONFIG.video, context: options })
|
|
176
|
-
}
|
|
177
|
-
|
|
178
|
-
// Notes for the replay: what this page check did and what it found, read on the task before anyone presses play
|
|
179
|
-
function pageNotes(result) {
|
|
180
|
-
const failedRequests = result.failedRequests.length + result.badResponses.length
|
|
181
|
-
const found = [
|
|
182
|
-
result.error && `the check stopped: ${result.error}`,
|
|
183
|
-
result.status >= 400 && `the page answered ${result.status}`,
|
|
184
|
-
result.pageErrors.length > 0 && `${result.pageErrors.length} page errors`,
|
|
185
|
-
result.consoleErrors.length > 0 && `${result.consoleErrors.length} console errors`,
|
|
186
|
-
failedRequests > 0 && `${failedRequests} failed requests`,
|
|
187
|
-
result.horizontalOverflowPx > 0 && `the page scrolls sideways by ${result.horizontalOverflowPx}px`,
|
|
188
|
-
].filter(Boolean)
|
|
189
|
-
const outcome = found.length > 0 ? `Found ${found.join(', ')}.` : 'No errors, failed requests or sideways scroll.'
|
|
190
|
-
return `Opened ${result.path} at ${result.width}px and ran the steps. ${outcome}`.slice(0, 600)
|
|
191
|
-
}
|
|
192
|
-
|
|
193
|
-
async function checkPage(browser, url, path, width, axeSource) {
|
|
194
|
-
const context = await openContext(browser, path, width)
|
|
195
|
-
const page = await context.newPage()
|
|
196
|
-
const mark = label => recorder?.mark(page, label)
|
|
197
|
-
const result = { path, width, consoleErrors: [], pageErrors: [], failedRequests: [], badResponses: [] }
|
|
198
|
-
const add = (list, item) => { if (list.length < MAX_ITEMS) list.push(item) }
|
|
199
|
-
page.on('console', message => { if (message.type() === 'error') add(result.consoleErrors, message.text()) })
|
|
200
|
-
page.on('pageerror', error => add(result.pageErrors, String(error.message)))
|
|
201
|
-
page.on('requestfailed', request => add(result.failedRequests, `${request.method()} ${request.url()} ${request.failure()?.errorText ?? ''}`))
|
|
202
|
-
page.on('response', response => { if (response.status() >= 400) add(result.badResponses, `${response.status()} ${response.url()}`) })
|
|
203
|
-
try {
|
|
204
|
-
const response = await page.goto(url, { waitUntil: 'load', timeout: CONFIG.navigationTimeoutMs })
|
|
205
|
-
result.status = response?.status() ?? null
|
|
206
|
-
await page.waitForLoadState('networkidle', { timeout: 5_000 }).catch(() => {})
|
|
207
|
-
await mark(`Opened ${path}`)
|
|
208
|
-
result.steps = await steps(page, { path, width, mark })
|
|
209
|
-
result.title = await page.title()
|
|
210
|
-
result.finalUrl = page.url()
|
|
211
|
-
result.horizontalOverflowPx = await page.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth)
|
|
212
|
-
const name = `${path.replace(/[^a-z0-9]+/gi, '_').replace(/^_|_$/g, '') || 'root'}-${width}.png`
|
|
213
|
-
result.screenshot = join(CONFIG.outDir, name)
|
|
214
|
-
await page.screenshot({ path: result.screenshot, fullPage: true })
|
|
215
|
-
if (axeSource) result.axe = await runAxe(page, axeSource)
|
|
216
|
-
} catch (error) {
|
|
217
|
-
result.error = String(error.message).split('\n')[0]
|
|
218
|
-
} finally {
|
|
219
|
-
result.recording = recorder ? await recorder.closeRecorded(context, { notes: pageNotes(result) }) : await context.close()
|
|
220
|
-
}
|
|
221
|
-
return result
|
|
222
|
-
}
|
|
223
|
-
|
|
224
|
-
async function main() {
|
|
225
|
-
mkdirSync(CONFIG.outDir, { recursive: true })
|
|
226
|
-
const summary = { pages: [], serverOutputTail: serverOutput }
|
|
227
|
-
const playwright = loadFromProject(['playwright', '@playwright/test', 'playwright-core'])
|
|
228
|
-
if (!playwright?.chromium) throw new Error(`Playwright is not installed in ${CONFIG.projectDir} and TAMSENSE_PLAYWRIGHT is not set`)
|
|
229
|
-
const axeSource = loadFromProject(['axe-core'])?.source ?? null
|
|
230
|
-
summary.axe = axeSource ? 'axe-core found in the project' : 'axe-core not installed, scan skipped'
|
|
231
|
-
|
|
232
|
-
let server = null
|
|
233
|
-
let browser = null
|
|
234
|
-
try {
|
|
235
|
-
let baseUrl = CONFIG.baseUrl
|
|
236
|
-
if (CONFIG.startCommand) {
|
|
237
|
-
const port = await freePort()
|
|
238
|
-
server = startServer(CONFIG.startCommand, port)
|
|
239
|
-
baseUrl = `http://localhost:${port}`
|
|
240
|
-
}
|
|
241
|
-
if (!baseUrl) throw new Error('set CONFIG.startCommand or CONFIG.baseUrl')
|
|
242
|
-
summary.baseUrl = baseUrl
|
|
243
|
-
await waitForServer(baseUrl, server)
|
|
244
|
-
browser = await launchBrowser(playwright.chromium)
|
|
245
|
-
for (const path of CONFIG.paths) {
|
|
246
|
-
for (const width of CONFIG.widths) {
|
|
247
|
-
summary.pages.push(await checkPage(browser, new URL(path, baseUrl).href, path, width, axeSource))
|
|
248
|
-
}
|
|
249
|
-
}
|
|
250
|
-
} catch (error) {
|
|
251
|
-
summary.setupError = String(error.message)
|
|
252
|
-
process.exitCode = 2
|
|
253
|
-
} finally {
|
|
254
|
-
await browser?.close().catch(() => {})
|
|
255
|
-
if (server) summary.server = stopServer(server)
|
|
256
|
-
writeFileSync(join(CONFIG.outDir, 'summary.json'), JSON.stringify(summary, null, 2))
|
|
257
|
-
console.log(JSON.stringify(summary, null, 2))
|
|
258
|
-
}
|
|
259
|
-
}
|
|
260
|
-
|
|
261
|
-
main().catch(error => {
|
|
262
|
-
console.error(error.message)
|
|
263
|
-
process.exitCode = 2
|
|
264
|
-
})
|
|
265
|
-
```
|
|
266
|
-
|
|
267
|
-
## Writing steps
|
|
268
|
-
|
|
269
|
-
`steps` runs on every page and width before the screenshot. Use it to reach the state the task changed: open the dialog, fill and submit the form, switch the tab. Branch on `path` when pages need different steps. Call `await mark('...')` after each meaningful step: the marks show on the replay's timeline for the people on the task. Whatever it returns is saved as `steps` in that page's summary (for example the focus order from a keyboard pass, see the `accessibility` skill). Use the same rules as in tests: role and label locators, web-first waits, no fixed sleeps (see [playwright-patterns.md](playwright-patterns.md)).
|
|
270
|
-
|
|
271
|
-
```js
|
|
272
|
-
async function steps(page, { path, mark }) {
|
|
273
|
-
if (path !== '/settings') return
|
|
274
|
-
await page.getByRole('button', { name: 'Edit profile' }).click()
|
|
275
|
-
await mark('Opened the profile form')
|
|
276
|
-
await page.getByLabel('Display name').fill('A very long display name that should wrap or truncate')
|
|
277
|
-
await page.getByRole('button', { name: 'Save' }).click()
|
|
278
|
-
await page.getByRole('status').waitFor()
|
|
279
|
-
await mark('Saved a long display name')
|
|
280
|
-
}
|
|
281
|
-
```
|
|
282
|
-
|
|
283
|
-
Steps that write data only touch the local app and whatever local database it uses. If the dev server is configured to talk to a shared or production backend or database, do not submit forms or trigger writes; check that first (see the `safe-database-changes` skill for how to read where the app connects).
|
|
284
|
-
|
|
285
|
-
## Reading the summary
|
|
286
|
-
|
|
287
|
-
- `server`: should be `stopped`. Anything else names a process you must stop yourself before finishing.
|
|
288
|
-
- `recording`: where each page's replay (and video) is, with the notes the script wrote from this summary. They are attached to the task when the run ends. When you know what a replay proves (the expected state showed, or exactly what broke), attach it sooner with `attach_replay` and your own notes: what you opened, clicked and expected, and whether it passed.
|
|
289
|
-
- `setupError`: the app or the browser did not start. Report it with the last lines of `serverOutputTail` (strip anything that looks like a secret). Do not retry with a different command unless the first one was clearly wrong, and never invent results.
|
|
290
|
-
- `status` 4xx or 5xx, `pageErrors`, `consoleErrors`: real findings when they come from the changed code; noise from browser extensions does not exist here, but third-party scripts and dev-server warnings do. Say which is which.
|
|
291
|
-
- `badResponses` and `failedRequests`: API calls the page needed that failed. A 401 or a redirect to a login page means the page needs a signed-in user (see the auth section of [playwright-patterns.md](playwright-patterns.md)).
|
|
292
|
-
- `horizontalOverflowPx` above 0 at 390 px: the page scrolls sideways on phones.
|
|
293
|
-
- `axe`: list each rule with impact and a target; the `accessibility` skill explains how to judge them.
|
|
294
|
-
|
|
295
|
-
Written for this library. The start, wait, act, stop approach follows anthropics/skills `webapp-testing` (Apache-2.0, commit dbd4588f9e1033efb41dad4bef2f7947c8993d44, Copyright 2026 Anthropic, PBC); the script is new Node.js code, not a translation of its Python helper.
|
|
1
|
+
# Headless check without an E2E suite
|
|
2
|
+
|
|
3
|
+
Read this when the project has no Playwright or Cypress tests you can run for the change, but you still need to see the pages: screenshots, console errors, failed requests, horizontal overflow on narrow screens, and optionally an axe scan.
|
|
4
|
+
|
|
5
|
+
The script below starts the app with the project's own command on a free port, waits for it, opens each page headless at a desktop and a phone width, runs optional steps you write, saves screenshots and a JSON summary to a temp folder, and stops the app and all its child processes at the end. In a TAMsense run each page and width is also recorded as a replay for the task, named from `CONFIG.title` and `CONFIG.purpose` (see [recording.md](recording.md)).
|
|
6
|
+
|
|
7
|
+
## What it needs
|
|
8
|
+
|
|
9
|
+
- Node.js 20 or newer.
|
|
10
|
+
- The `playwright`, `@playwright/test` or `playwright-core` package already installed in the project (it is resolved from the project folder you give it; in a monorepo, use the package folder of the web app). If the project has none and the `TAMSENSE_PLAYWRIGHT` environment variable is set, the script loads the `playwright-core` that the TAMsense runner installed in that folder, with the headless browser from the `TAMSENSE_PLAYWRIGHT_BROWSERS` folder. Nothing is installed.
|
|
11
|
+
- A browser that is already on the machine: Playwright's downloaded Chromium (the project's or the runner's), or the system Chrome or Edge as a fallback. If none works, the script stops and says so. Do not run `playwright install` or add packages to get one; report that the check could not run.
|
|
12
|
+
- `axe-core` is optional: if the project already has it, each page also gets an accessibility scan.
|
|
13
|
+
|
|
14
|
+
## How to use it
|
|
15
|
+
|
|
16
|
+
1. Make a new folder in the OS temp directory (`node -p "require('os').tmpdir()"`), for example `<tmp>/ui-check-<random>`. Never inside the repository.
|
|
17
|
+
2. Write the script there as `ui-check.mjs` and edit only the `CONFIG` block and the `steps` function. Set `title` and `purpose` to what this check is and why ("Feature testing, attempt 2", "Checking the empty orders state after the fix"), and `video: true` when the pages have a canvas, a map, a video player or iframes from another site.
|
|
18
|
+
3. Run `node <tmp>/ui-check-<random>/ui-check.mjs` with a timeout of a few minutes. It prints the summary; screenshots are in `CONFIG.outDir`.
|
|
19
|
+
4. Open the screenshots with the Read tool and look at them. Describe in the report what each one shows; the paths alone mean nothing to the reader.
|
|
20
|
+
5. Delete the folder when the report is written, unless the task asks you to keep artifacts.
|
|
21
|
+
|
|
22
|
+
`startCommand` is the project's own dev or preview command with `{port}` where the port goes, for example `pnpm dev --port {port} --strictPort` (Vite), `npm run dev -- -p {port}` (Next.js), `pnpm nuxt dev --port {port}`, `bin/rails server -p {port}`, `python manage.py runserver 127.0.0.1:{port}`. The script also sets `PORT`. Leave it empty and set `baseUrl` if the app is already running or is static files you serve another way.
|
|
23
|
+
|
|
24
|
+
On Windows the command runs in `cmd.exe`, which may not see the same PATH as your shell. If the summary says the command is not recognized, use absolute paths (for example the full path to `node.exe` and the package manager's script) rather than changing anything in the project.
|
|
25
|
+
|
|
26
|
+
## The script
|
|
27
|
+
|
|
28
|
+
```js
|
|
29
|
+
import { createRequire } from 'node:module'
|
|
30
|
+
import { spawn, execFileSync } from 'node:child_process'
|
|
31
|
+
import { mkdirSync, writeFileSync } from 'node:fs'
|
|
32
|
+
import { join } from 'node:path'
|
|
33
|
+
import net from 'node:net'
|
|
34
|
+
|
|
35
|
+
const CONFIG = {
|
|
36
|
+
projectDir: '/absolute/path/to/web/app',
|
|
37
|
+
outDir: '/absolute/path/to/tmp/ui-check-1234/out',
|
|
38
|
+
startCommand: 'pnpm dev --port {port} --strictPort',
|
|
39
|
+
baseUrl: '',
|
|
40
|
+
paths: ['/'],
|
|
41
|
+
widths: [1280, 390],
|
|
42
|
+
startTimeoutMs: 120_000,
|
|
43
|
+
navigationTimeoutMs: 30_000,
|
|
44
|
+
title: 'Feature testing',
|
|
45
|
+
purpose: 'Checking the changed screens at desktop and phone width',
|
|
46
|
+
video: false,
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
async function steps(page, { path, width, mark }) {
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const MAX_ITEMS = 20
|
|
53
|
+
const MAX_TITLE = 60
|
|
54
|
+
const serverOutput = []
|
|
55
|
+
const recorder = process.env.TAMSENSE_RECORDER ? createRequire(import.meta.url)(process.env.TAMSENSE_RECORDER) : null
|
|
56
|
+
|
|
57
|
+
function keepTail(chunk) {
|
|
58
|
+
serverOutput.push(...String(chunk).split('\n').filter(Boolean))
|
|
59
|
+
serverOutput.splice(0, Math.max(0, serverOutput.length - 40))
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
function requireFrom(dir, names) {
|
|
63
|
+
const require = createRequire(join(dir, 'package.json'))
|
|
64
|
+
for (const name of names) {
|
|
65
|
+
try {
|
|
66
|
+
return require(name)
|
|
67
|
+
} catch {}
|
|
68
|
+
}
|
|
69
|
+
return null
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
function loadFromProject(names) {
|
|
73
|
+
const found = requireFrom(CONFIG.projectDir, names)
|
|
74
|
+
const runnerCopy = process.env.TAMSENSE_PLAYWRIGHT
|
|
75
|
+
if (found || !names.includes('playwright-core') || !runnerCopy) return found
|
|
76
|
+
if (process.env.TAMSENSE_PLAYWRIGHT_BROWSERS) process.env.PLAYWRIGHT_BROWSERS_PATH = process.env.TAMSENSE_PLAYWRIGHT_BROWSERS
|
|
77
|
+
return requireFrom(runnerCopy, ['playwright-core'])
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
function freePort() {
|
|
81
|
+
return new Promise((resolve, reject) => {
|
|
82
|
+
const server = net.createServer()
|
|
83
|
+
server.on('error', reject)
|
|
84
|
+
server.listen(0, '127.0.0.1', () => {
|
|
85
|
+
const { port } = server.address()
|
|
86
|
+
server.close(() => resolve(port))
|
|
87
|
+
})
|
|
88
|
+
})
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
function startServer(command, port) {
|
|
92
|
+
const child = spawn(command.replaceAll('{port}', String(port)), {
|
|
93
|
+
cwd: CONFIG.projectDir,
|
|
94
|
+
shell: true,
|
|
95
|
+
env: { ...process.env, PORT: String(port), BROWSER: 'none' },
|
|
96
|
+
detached: process.platform !== 'win32',
|
|
97
|
+
windowsHide: true,
|
|
98
|
+
stdio: ['ignore', 'pipe', 'pipe'],
|
|
99
|
+
})
|
|
100
|
+
child.stdout.on('data', keepTail)
|
|
101
|
+
child.stderr.on('data', keepTail)
|
|
102
|
+
return child
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
function stopServer(child) {
|
|
106
|
+
if (!child || child.exitCode !== null) return 'not running'
|
|
107
|
+
try {
|
|
108
|
+
if (process.platform === 'win32') {
|
|
109
|
+
const taskkill = join(process.env.SystemRoot ?? 'C:\\Windows', 'System32', 'taskkill.exe')
|
|
110
|
+
execFileSync(taskkill, ['/pid', String(child.pid), '/T', '/F'], { stdio: 'ignore' })
|
|
111
|
+
} else {
|
|
112
|
+
process.kill(-child.pid, 'SIGKILL')
|
|
113
|
+
}
|
|
114
|
+
return 'stopped'
|
|
115
|
+
} catch (error) {
|
|
116
|
+
child.kill('SIGKILL')
|
|
117
|
+
return `could not stop the process tree of pid ${child.pid}: ${error.message}`
|
|
118
|
+
} finally {
|
|
119
|
+
child.stdout.destroy()
|
|
120
|
+
child.stderr.destroy()
|
|
121
|
+
child.unref()
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
async function waitForServer(url, child) {
|
|
126
|
+
const deadline = Date.now() + CONFIG.startTimeoutMs
|
|
127
|
+
while (Date.now() < deadline) {
|
|
128
|
+
if (child && child.exitCode !== null) throw new Error(`start command exited with code ${child.exitCode}`)
|
|
129
|
+
try {
|
|
130
|
+
await fetch(url, { redirect: 'manual', signal: AbortSignal.timeout(5_000) })
|
|
131
|
+
return
|
|
132
|
+
} catch {}
|
|
133
|
+
await new Promise(resolve => setTimeout(resolve, 1_000))
|
|
134
|
+
}
|
|
135
|
+
throw new Error(`no response from ${url} within ${CONFIG.startTimeoutMs / 1000}s`)
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
async function launchBrowser(chromium) {
|
|
139
|
+
const failures = []
|
|
140
|
+
for (const channel of [undefined, 'chromium', 'chrome', 'msedge']) {
|
|
141
|
+
try {
|
|
142
|
+
return await chromium.launch({ headless: true, channel })
|
|
143
|
+
} catch (error) {
|
|
144
|
+
failures.push(`${channel ?? 'playwright chromium'}: ${String(error.message).split('\n')[0]}`)
|
|
145
|
+
}
|
|
146
|
+
}
|
|
147
|
+
throw new Error(`no browser available without downloading one: ${failures.join(' | ')}`)
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
async function runAxe(page, axeSource) {
|
|
151
|
+
try {
|
|
152
|
+
await page.addScriptTag({ content: axeSource })
|
|
153
|
+
return await page.evaluate(async () => {
|
|
154
|
+
const result = await window.axe.run(document, {
|
|
155
|
+
runOnly: { type: 'tag', values: ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa'] },
|
|
156
|
+
})
|
|
157
|
+
return result.violations.map(violation => ({
|
|
158
|
+
rule: violation.id,
|
|
159
|
+
impact: violation.impact,
|
|
160
|
+
help: violation.help,
|
|
161
|
+
count: violation.nodes.length,
|
|
162
|
+
targets: violation.nodes.slice(0, 5).map(node => JSON.stringify(node.target)),
|
|
163
|
+
}))
|
|
164
|
+
})
|
|
165
|
+
} catch (error) {
|
|
166
|
+
return { error: String(error.message).split('\n')[0] }
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
async function openContext(browser, path, width) {
|
|
171
|
+
const options = { viewport: { width, height: 900 } }
|
|
172
|
+
if (!recorder) return browser.newContext(options)
|
|
173
|
+
const where = `, ${path} at ${width}px`
|
|
174
|
+
const title = `${CONFIG.title.slice(0, MAX_TITLE - where.length)}${where}`
|
|
175
|
+
return recorder.newRecordedContext(browser, { title, purpose: CONFIG.purpose, video: CONFIG.video, context: options })
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
// Notes for the replay: what this page check did and what it found, read on the task before anyone presses play
|
|
179
|
+
function pageNotes(result) {
|
|
180
|
+
const failedRequests = result.failedRequests.length + result.badResponses.length
|
|
181
|
+
const found = [
|
|
182
|
+
result.error && `the check stopped: ${result.error}`,
|
|
183
|
+
result.status >= 400 && `the page answered ${result.status}`,
|
|
184
|
+
result.pageErrors.length > 0 && `${result.pageErrors.length} page errors`,
|
|
185
|
+
result.consoleErrors.length > 0 && `${result.consoleErrors.length} console errors`,
|
|
186
|
+
failedRequests > 0 && `${failedRequests} failed requests`,
|
|
187
|
+
result.horizontalOverflowPx > 0 && `the page scrolls sideways by ${result.horizontalOverflowPx}px`,
|
|
188
|
+
].filter(Boolean)
|
|
189
|
+
const outcome = found.length > 0 ? `Found ${found.join(', ')}.` : 'No errors, failed requests or sideways scroll.'
|
|
190
|
+
return `Opened ${result.path} at ${result.width}px and ran the steps. ${outcome}`.slice(0, 600)
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
async function checkPage(browser, url, path, width, axeSource) {
|
|
194
|
+
const context = await openContext(browser, path, width)
|
|
195
|
+
const page = await context.newPage()
|
|
196
|
+
const mark = label => recorder?.mark(page, label)
|
|
197
|
+
const result = { path, width, consoleErrors: [], pageErrors: [], failedRequests: [], badResponses: [] }
|
|
198
|
+
const add = (list, item) => { if (list.length < MAX_ITEMS) list.push(item) }
|
|
199
|
+
page.on('console', message => { if (message.type() === 'error') add(result.consoleErrors, message.text()) })
|
|
200
|
+
page.on('pageerror', error => add(result.pageErrors, String(error.message)))
|
|
201
|
+
page.on('requestfailed', request => add(result.failedRequests, `${request.method()} ${request.url()} ${request.failure()?.errorText ?? ''}`))
|
|
202
|
+
page.on('response', response => { if (response.status() >= 400) add(result.badResponses, `${response.status()} ${response.url()}`) })
|
|
203
|
+
try {
|
|
204
|
+
const response = await page.goto(url, { waitUntil: 'load', timeout: CONFIG.navigationTimeoutMs })
|
|
205
|
+
result.status = response?.status() ?? null
|
|
206
|
+
await page.waitForLoadState('networkidle', { timeout: 5_000 }).catch(() => {})
|
|
207
|
+
await mark(`Opened ${path}`)
|
|
208
|
+
result.steps = await steps(page, { path, width, mark })
|
|
209
|
+
result.title = await page.title()
|
|
210
|
+
result.finalUrl = page.url()
|
|
211
|
+
result.horizontalOverflowPx = await page.evaluate(() => document.documentElement.scrollWidth - document.documentElement.clientWidth)
|
|
212
|
+
const name = `${path.replace(/[^a-z0-9]+/gi, '_').replace(/^_|_$/g, '') || 'root'}-${width}.png`
|
|
213
|
+
result.screenshot = join(CONFIG.outDir, name)
|
|
214
|
+
await page.screenshot({ path: result.screenshot, fullPage: true })
|
|
215
|
+
if (axeSource) result.axe = await runAxe(page, axeSource)
|
|
216
|
+
} catch (error) {
|
|
217
|
+
result.error = String(error.message).split('\n')[0]
|
|
218
|
+
} finally {
|
|
219
|
+
result.recording = recorder ? await recorder.closeRecorded(context, { notes: pageNotes(result) }) : await context.close()
|
|
220
|
+
}
|
|
221
|
+
return result
|
|
222
|
+
}
|
|
223
|
+
|
|
224
|
+
async function main() {
|
|
225
|
+
mkdirSync(CONFIG.outDir, { recursive: true })
|
|
226
|
+
const summary = { pages: [], serverOutputTail: serverOutput }
|
|
227
|
+
const playwright = loadFromProject(['playwright', '@playwright/test', 'playwright-core'])
|
|
228
|
+
if (!playwright?.chromium) throw new Error(`Playwright is not installed in ${CONFIG.projectDir} and TAMSENSE_PLAYWRIGHT is not set`)
|
|
229
|
+
const axeSource = loadFromProject(['axe-core'])?.source ?? null
|
|
230
|
+
summary.axe = axeSource ? 'axe-core found in the project' : 'axe-core not installed, scan skipped'
|
|
231
|
+
|
|
232
|
+
let server = null
|
|
233
|
+
let browser = null
|
|
234
|
+
try {
|
|
235
|
+
let baseUrl = CONFIG.baseUrl
|
|
236
|
+
if (CONFIG.startCommand) {
|
|
237
|
+
const port = await freePort()
|
|
238
|
+
server = startServer(CONFIG.startCommand, port)
|
|
239
|
+
baseUrl = `http://localhost:${port}`
|
|
240
|
+
}
|
|
241
|
+
if (!baseUrl) throw new Error('set CONFIG.startCommand or CONFIG.baseUrl')
|
|
242
|
+
summary.baseUrl = baseUrl
|
|
243
|
+
await waitForServer(baseUrl, server)
|
|
244
|
+
browser = await launchBrowser(playwright.chromium)
|
|
245
|
+
for (const path of CONFIG.paths) {
|
|
246
|
+
for (const width of CONFIG.widths) {
|
|
247
|
+
summary.pages.push(await checkPage(browser, new URL(path, baseUrl).href, path, width, axeSource))
|
|
248
|
+
}
|
|
249
|
+
}
|
|
250
|
+
} catch (error) {
|
|
251
|
+
summary.setupError = String(error.message)
|
|
252
|
+
process.exitCode = 2
|
|
253
|
+
} finally {
|
|
254
|
+
await browser?.close().catch(() => {})
|
|
255
|
+
if (server) summary.server = stopServer(server)
|
|
256
|
+
writeFileSync(join(CONFIG.outDir, 'summary.json'), JSON.stringify(summary, null, 2))
|
|
257
|
+
console.log(JSON.stringify(summary, null, 2))
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
main().catch(error => {
|
|
262
|
+
console.error(error.message)
|
|
263
|
+
process.exitCode = 2
|
|
264
|
+
})
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
## Writing steps
|
|
268
|
+
|
|
269
|
+
`steps` runs on every page and width before the screenshot. Use it to reach the state the task changed: open the dialog, fill and submit the form, switch the tab. Branch on `path` when pages need different steps. Call `await mark('...')` after each meaningful step: the marks show on the replay's timeline for the people on the task. Whatever it returns is saved as `steps` in that page's summary (for example the focus order from a keyboard pass, see the `accessibility` skill). Use the same rules as in tests: role and label locators, web-first waits, no fixed sleeps (see [playwright-patterns.md](playwright-patterns.md)).
|
|
270
|
+
|
|
271
|
+
```js
|
|
272
|
+
async function steps(page, { path, mark }) {
|
|
273
|
+
if (path !== '/settings') return
|
|
274
|
+
await page.getByRole('button', { name: 'Edit profile' }).click()
|
|
275
|
+
await mark('Opened the profile form')
|
|
276
|
+
await page.getByLabel('Display name').fill('A very long display name that should wrap or truncate')
|
|
277
|
+
await page.getByRole('button', { name: 'Save' }).click()
|
|
278
|
+
await page.getByRole('status').waitFor()
|
|
279
|
+
await mark('Saved a long display name')
|
|
280
|
+
}
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
Steps that write data only touch the local app and whatever local database it uses. If the dev server is configured to talk to a shared or production backend or database, do not submit forms or trigger writes; check that first (see the `safe-database-changes` skill for how to read where the app connects).
|
|
284
|
+
|
|
285
|
+
## Reading the summary
|
|
286
|
+
|
|
287
|
+
- `server`: should be `stopped`. Anything else names a process you must stop yourself before finishing.
|
|
288
|
+
- `recording`: where each page's replay (and video) is, with the notes the script wrote from this summary. They are attached to the task when the run ends. When you know what a replay proves (the expected state showed, or exactly what broke), attach it sooner with `attach_replay` and your own notes: what you opened, clicked and expected, and whether it passed.
|
|
289
|
+
- `setupError`: the app or the browser did not start. Report it with the last lines of `serverOutputTail` (strip anything that looks like a secret). Do not retry with a different command unless the first one was clearly wrong, and never invent results.
|
|
290
|
+
- `status` 4xx or 5xx, `pageErrors`, `consoleErrors`: real findings when they come from the changed code; noise from browser extensions does not exist here, but third-party scripts and dev-server warnings do. Say which is which.
|
|
291
|
+
- `badResponses` and `failedRequests`: API calls the page needed that failed. A 401 or a redirect to a login page means the page needs a signed-in user (see the auth section of [playwright-patterns.md](playwright-patterns.md)).
|
|
292
|
+
- `horizontalOverflowPx` above 0 at 390 px: the page scrolls sideways on phones.
|
|
293
|
+
- `axe`: list each rule with impact and a target; the `accessibility` skill explains how to judge them.
|
|
294
|
+
|
|
295
|
+
Written for this library. The start, wait, act, stop approach follows anthropics/skills `webapp-testing` (Apache-2.0, commit dbd4588f9e1033efb41dad4bef2f7947c8993d44, Copyright 2026 Anthropic, PBC); the script is new Node.js code, not a translation of its Python helper.
|