@tamsensedev/runner 0.10.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +5 -5
- package/README.md +61 -50
- package/dist/cli.js +1188 -324
- package/dist/cli.js.map +1 -1
- package/dist/recorder.cjs +42 -6
- package/dist/recorder.cjs.map +1 -1
- package/library/.claude-plugin/plugin.json +5 -5
- package/library/THIRD_PARTY_NOTICES.md +123 -123
- package/library/agents/code-architect.md +67 -67
- package/library/agents/code-explorer.md +61 -61
- package/library/agents/code-reviewer.md +73 -73
- package/library/agents/code-simplifier.md +62 -62
- package/library/agents/design-critic.md +52 -52
- package/library/agents/second-opinion.md +64 -64
- package/library/agents/security-auditor.md +86 -86
- package/library/agents/silent-failure-hunter.md +70 -70
- package/library/agents/test-coverage-analyzer.md +78 -78
- package/library/agents/test-runner.md +48 -48
- package/library/agents/ui-tester.md +50 -50
- package/library/licenses/Apache-2.0.txt +202 -202
- package/library/skills/accessibility/SKILL.md +144 -144
- package/library/skills/accessibility/references/wcag-checks.md +168 -168
- package/library/skills/addressing-review-feedback/SKILL.md +109 -109
- package/library/skills/app-screens/SKILL.md +96 -96
- package/library/skills/app-screens/references/shell-kpis-charts.md +108 -108
- package/library/skills/app-screens/references/states-density-phone.md +54 -54
- package/library/skills/app-screens/references/tables-forms-settings.md +104 -104
- package/library/skills/avoiding-generic-ai-design/SKILL.md +92 -92
- package/library/skills/avoiding-generic-ai-design/references/tells.md +112 -112
- package/library/skills/clean-commits/SKILL.md +85 -85
- package/library/skills/design-review/SKILL.md +107 -107
- package/library/skills/design-review/references/critique-passes.md +101 -101
- package/library/skills/design-review/references/screenshot-procedure.md +121 -121
- package/library/skills/differential-security-review/LICENSE +427 -427
- package/library/skills/differential-security-review/SKILL.md +105 -105
- package/library/skills/differential-security-review/references/adversarial.md +115 -115
- package/library/skills/differential-security-review/references/methodology.md +145 -145
- package/library/skills/differential-security-review/references/patterns.md +108 -108
- package/library/skills/false-positive-check/LICENSE +427 -427
- package/library/skills/false-positive-check/SKILL.md +95 -95
- package/library/skills/false-positive-check/references/bug-class-verification.md +128 -128
- package/library/skills/false-positive-check/references/deep-verification.md +113 -113
- package/library/skills/false-positive-check/references/evidence-templates.md +93 -93
- package/library/skills/false-positive-check/references/false-positive-patterns.md +122 -122
- package/library/skills/false-positive-check/references/gate-reviews.md +36 -36
- package/library/skills/false-positive-check/references/standard-verification.md +80 -80
- package/library/skills/focused-implementation/SKILL.md +104 -104
- package/library/skills/implementation-plan/SKILL.md +66 -66
- package/library/skills/interface-design/SKILL.md +144 -144
- package/library/skills/interface-design/references/craft-defaults.md +130 -130
- package/library/skills/interface-design/references/new-direction.md +62 -62
- package/library/skills/interface-design/references/writing.md +52 -52
- package/library/skills/interface-polish/SKILL.md +103 -103
- package/library/skills/interface-polish/references/detail-rules.md +154 -154
- package/library/skills/interface-polish/references/worst-case-data.md +158 -158
- package/library/skills/motion-design/SKILL.md +106 -106
- package/library/skills/motion-design/references/recipes.md +243 -243
- package/library/skills/postgres-performance/SKILL.md +97 -97
- package/library/skills/postgres-performance/references/rules.md +265 -265
- package/library/skills/project-orientation/SKILL.md +77 -77
- package/library/skills/reviewing-changes/SKILL.md +183 -183
- package/library/skills/safe-database-changes/SKILL.md +111 -111
- package/library/skills/safe-database-changes/references/backfills.md +75 -75
- package/library/skills/safe-database-changes/references/migration-tools.md +69 -69
- package/library/skills/safe-database-changes/references/postgres-ddl.md +199 -199
- package/library/skills/security-review/SKILL.md +132 -132
- package/library/skills/security-review/references/checks-by-area.md +107 -107
- package/library/skills/security-review/references/do-not-report.md +86 -86
- package/library/skills/spec-writing/SKILL.md +144 -144
- package/library/skills/systematic-debugging/SKILL.md +130 -130
- package/library/skills/systematic-debugging/references/flaky-tests.md +51 -51
- package/library/skills/systematic-debugging/references/root-cause-tracing.md +55 -55
- package/library/skills/task-interview/SKILL.md +84 -84
- package/library/skills/test-driven-development/SKILL.md +116 -116
- package/library/skills/test-driven-development/references/writing-good-tests.md +85 -85
- package/library/skills/ui-testing/SKILL.md +102 -102
- package/library/skills/ui-testing/references/headless-check.md +295 -280
- package/library/skills/ui-testing/references/local-stand.md +51 -51
- package/library/skills/ui-testing/references/playwright-patterns.md +134 -134
- package/library/skills/ui-testing/references/recording.md +61 -58
- package/library/skills/verification-before-completion/SKILL.md +113 -113
- package/library/skills/web-interface-guidelines/SKILL.md +202 -202
- package/package.json +46 -46
|
@@ -1,51 +1,51 @@
|
|
|
1
|
-
# Starting a local copy of the app
|
|
2
|
-
|
|
3
|
-
Read this before you start the app for a check. In a TAMsense run the task folder is a fresh checkout: no dependencies, no `.env`, nothing running. You set up a local copy from this checkout yourself, every time, and you take it down at the end. The person's own setup, their databases and every shared server stay untouched.
|
|
4
|
-
|
|
5
|
-
## The rules
|
|
6
|
-
|
|
7
|
-
1. **This task folder only.** Everything you start comes from the checkout in the task folder. Never read or copy `.env` files, keys or settings from elsewhere on this computer, including the person's own checkout of the same project.
|
|
8
|
-
2. **Find the local setup yourself.** Look in the README, compose files named local or dev (`docker-compose.local.yml`, `infra/compose/local.compose.yml`, `compose.dev.yaml`), `.env.example` and other env files meant for a local run (`.env.local.example`, `.env.development`), and the package scripts (`dev`, `db:migrate`, `db:seed`). The CI workflow often shows the exact services and commands.
|
|
9
|
-
3. **Check every address before you start.** In the settings you use, databases, caches, queues and APIs must point at `localhost`, `127.0.0.1` or containers you start. A staging, shared dev or production address, or a cloud host you do not start, means stop (rule 7). Keys of outside services (payments, mail, analytics, storage, AI APIs) stay empty; never fill them with real ones. Check variables by name and address only; never print secrets.
|
|
10
|
-
4. **Your own compose project on free ports.** Start containers as a separate compose project named `tamsense-task-<task id>`, so they never mix with the person's containers. Map them to free host ports with an override file in a temp folder, not by editing the project's files. Find a free port by asking the system (`node -e "const s=require('net').createServer().listen(0,()=>{console.log(s.address().port);s.close()})"`).
|
|
11
|
-
5. **Never touch containers you did not start.** Do not stop, restart, remove or change other containers, volumes or networks, even when a port is taken: pick another port.
|
|
12
|
-
6. **Clean up.** At the end run `docker compose -p tamsense-task-<task id> down -v` with the same files, stop the app by its process id, and delete the temp folder.
|
|
13
|
-
7. **When it cannot run, stop.** If Docker is not running, the project has no local setup, or the settings point outside this computer, do not start anything and do not look for a workaround. In a check step, call `submit_verdict` with `passed: false` and `couldNotRun: true`, write the exact reason and what a person should do in the report ("Docker is not running: start Docker Desktop and run this step again"), and do not hand off. The task then waits on this step for a person. In a code step, say so in your report.
|
|
14
|
-
|
|
15
|
-
The browser adds a guard of its own: requests other than GET, HEAD and OPTIONS to an address outside this computer are aborted (see [recording.md](recording.md)). It does not replace rule 3: the app's server talks to its database and services directly.
|
|
16
|
-
|
|
17
|
-
## Example
|
|
18
|
-
|
|
19
|
-
Task 40, a project with `infra/compose/local.compose.yml` (postgres and redis), `.env.example` pointing at `localhost:5432` and `localhost:6379`, and a `dev` script.
|
|
20
|
-
|
|
21
|
-
```bash
|
|
22
|
-
docker info --format '{{.ServerVersion}}' # an error here means Docker is not running: stop (rule 7)
|
|
23
|
-
TMP=$(node -p "require('fs').mkdtempSync(require('path').join(require('os').tmpdir(), 'tamsense-task-40-'))")
|
|
24
|
-
|
|
25
|
-
# free host ports for the two services, found as in rule 4
|
|
26
|
-
cat > "$TMP/ports.override.yml" <<'EOF'
|
|
27
|
-
services:
|
|
28
|
-
postgres:
|
|
29
|
-
ports: !override
|
|
30
|
-
- "55432:5432"
|
|
31
|
-
redis:
|
|
32
|
-
ports: !override
|
|
33
|
-
- "56379:6379"
|
|
34
|
-
EOF
|
|
35
|
-
|
|
36
|
-
docker compose -p tamsense-task-40 -f infra/compose/local.compose.yml -f "$TMP/ports.override.yml" up -d postgres redis
|
|
37
|
-
|
|
38
|
-
pnpm install --frozen-lockfile
|
|
39
|
-
# settings from .env.example, with the ports changed to the free ones, passed as variables, not written into the repository
|
|
40
|
-
export DATABASE_URL=postgresql://postgres:postgres@localhost:55432/app REDIS_URL=redis://localhost:56379
|
|
41
|
-
pnpm db:migrate # migrations go to the local database only
|
|
42
|
-
BROWSER=none pnpm dev --port 53100 # the project's own dev script on a free port; note its process id
|
|
43
|
-
|
|
44
|
-
# ... the headless check against http://localhost:53100 ...
|
|
45
|
-
|
|
46
|
-
# stop the dev server by its process id, then
|
|
47
|
-
docker compose -p tamsense-task-40 -f infra/compose/local.compose.yml -f "$TMP/ports.override.yml" down -v
|
|
48
|
-
rm -rf "$TMP"
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
Before `db:migrate`, look at the address once more: `DATABASE_URL` must name `localhost` and the port of your container. `!override` (Docker Compose 2.24 or newer) replaces the project's own port list instead of adding to it, so nothing binds the ports the person's own containers may use.
|
|
1
|
+
# Starting a local copy of the app
|
|
2
|
+
|
|
3
|
+
Read this before you start the app for a check. In a TAMsense run the task folder is a fresh checkout: no dependencies, no `.env`, nothing running. You set up a local copy from this checkout yourself, every time, and you take it down at the end. The person's own setup, their databases and every shared server stay untouched.
|
|
4
|
+
|
|
5
|
+
## The rules
|
|
6
|
+
|
|
7
|
+
1. **This task folder only.** Everything you start comes from the checkout in the task folder. Never read or copy `.env` files, keys or settings from elsewhere on this computer, including the person's own checkout of the same project.
|
|
8
|
+
2. **Find the local setup yourself.** Look in the README, compose files named local or dev (`docker-compose.local.yml`, `infra/compose/local.compose.yml`, `compose.dev.yaml`), `.env.example` and other env files meant for a local run (`.env.local.example`, `.env.development`), and the package scripts (`dev`, `db:migrate`, `db:seed`). The CI workflow often shows the exact services and commands.
|
|
9
|
+
3. **Check every address before you start.** In the settings you use, databases, caches, queues and APIs must point at `localhost`, `127.0.0.1` or containers you start. A staging, shared dev or production address, or a cloud host you do not start, means stop (rule 7). Keys of outside services (payments, mail, analytics, storage, AI APIs) stay empty; never fill them with real ones. Check variables by name and address only; never print secrets.
|
|
10
|
+
4. **Your own compose project on free ports.** Start containers as a separate compose project named `tamsense-task-<task id>`, so they never mix with the person's containers. Map them to free host ports with an override file in a temp folder, not by editing the project's files. Find a free port by asking the system (`node -e "const s=require('net').createServer().listen(0,()=>{console.log(s.address().port);s.close()})"`).
|
|
11
|
+
5. **Never touch containers you did not start.** Do not stop, restart, remove or change other containers, volumes or networks, even when a port is taken: pick another port.
|
|
12
|
+
6. **Clean up.** At the end run `docker compose -p tamsense-task-<task id> down -v` with the same files, stop the app by its process id, and delete the temp folder.
|
|
13
|
+
7. **When it cannot run, stop.** If Docker is not running, the project has no local setup, or the settings point outside this computer, do not start anything and do not look for a workaround. In a check step, call `submit_verdict` with `passed: false` and `couldNotRun: true`, write the exact reason and what a person should do in the report ("Docker is not running: start Docker Desktop and run this step again"), and do not hand off. The task then waits on this step for a person. In a code step, say so in your report.
|
|
14
|
+
|
|
15
|
+
The browser adds a guard of its own: requests other than GET, HEAD and OPTIONS to an address outside this computer are aborted (see [recording.md](recording.md)). It does not replace rule 3: the app's server talks to its database and services directly.
|
|
16
|
+
|
|
17
|
+
## Example
|
|
18
|
+
|
|
19
|
+
Task 40, a project with `infra/compose/local.compose.yml` (postgres and redis), `.env.example` pointing at `localhost:5432` and `localhost:6379`, and a `dev` script.
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
docker info --format '{{.ServerVersion}}' # an error here means Docker is not running: stop (rule 7)
|
|
23
|
+
TMP=$(node -p "require('fs').mkdtempSync(require('path').join(require('os').tmpdir(), 'tamsense-task-40-'))")
|
|
24
|
+
|
|
25
|
+
# free host ports for the two services, found as in rule 4
|
|
26
|
+
cat > "$TMP/ports.override.yml" <<'EOF'
|
|
27
|
+
services:
|
|
28
|
+
postgres:
|
|
29
|
+
ports: !override
|
|
30
|
+
- "55432:5432"
|
|
31
|
+
redis:
|
|
32
|
+
ports: !override
|
|
33
|
+
- "56379:6379"
|
|
34
|
+
EOF
|
|
35
|
+
|
|
36
|
+
docker compose -p tamsense-task-40 -f infra/compose/local.compose.yml -f "$TMP/ports.override.yml" up -d postgres redis
|
|
37
|
+
|
|
38
|
+
pnpm install --frozen-lockfile
|
|
39
|
+
# settings from .env.example, with the ports changed to the free ones, passed as variables, not written into the repository
|
|
40
|
+
export DATABASE_URL=postgresql://postgres:postgres@localhost:55432/app REDIS_URL=redis://localhost:56379
|
|
41
|
+
pnpm db:migrate # migrations go to the local database only
|
|
42
|
+
BROWSER=none pnpm dev --port 53100 # the project's own dev script on a free port; note its process id
|
|
43
|
+
|
|
44
|
+
# ... the headless check against http://localhost:53100 ...
|
|
45
|
+
|
|
46
|
+
# stop the dev server by its process id, then
|
|
47
|
+
docker compose -p tamsense-task-40 -f infra/compose/local.compose.yml -f "$TMP/ports.override.yml" down -v
|
|
48
|
+
rm -rf "$TMP"
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Before `db:migrate`, look at the address once more: `DATABASE_URL` must name `localhost` and the port of your container. `!override` (Docker Compose 2.24 or newer) replaces the project's own port list instead of adding to it, so nothing binds the ports the person's own containers may use.
|
|
@@ -1,134 +1,134 @@
|
|
|
1
|
-
# Playwright patterns: locators, waiting, flakes, auth, mocking
|
|
2
|
-
|
|
3
|
-
Read this when you write, fix or review browser tests or the `steps` of a headless check. In a TAMsense run every browser context, also those of the project's tests, is recorded for the task; scripts of your own name their recordings as [recording.md](recording.md) shows. Examples are Playwright Test in TypeScript; translate to the project's language and runner. If the project uses Cypress, the same ideas apply (see the last section). Check the installed version (`npx playwright --version`) before using an API you are not sure exists in it.
|
|
4
|
-
|
|
5
|
-
## Locators
|
|
6
|
-
|
|
7
|
-
Use the first one that works, in this order:
|
|
8
|
-
|
|
9
|
-
```ts
|
|
10
|
-
page.getByRole('button', { name: 'Save changes' }) // 1. role + accessible name
|
|
11
|
-
page.getByLabel('Email address') // 2. form fields by label
|
|
12
|
-
page.getByText('Welcome back') // 3. non-interactive text
|
|
13
|
-
page.getByPlaceholder('Search') // 4. only if there is no label
|
|
14
|
-
page.getByAltText('Company logo') // 5. images
|
|
15
|
-
page.getByTestId('checkout-summary') // 6. when nothing semantic is stable
|
|
16
|
-
page.locator('.legacy-widget button') // 7. CSS or XPath, last resort
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
- Role locators survive class renames and component swaps, and they fail when the control is not accessible, which is a finding in itself.
|
|
20
|
-
- `exact: true` stops "Log" from matching "Log out".
|
|
21
|
-
- Scope instead of indexing: `page.getByRole('dialog', { name: 'Delete project' }).getByRole('button', { name: 'Delete' })` beats `.nth(2)`.
|
|
22
|
-
- Rows and cards: `page.getByRole('row').filter({ hasText: 'Invoice 42' }).getByRole('button', { name: 'Open' })`.
|
|
23
|
-
- Use the project's existing `data-testid` convention if it has one; do not add test ids to product code during a check step.
|
|
24
|
-
- Strict mode errors ("resolved to 2 elements") mean the locator is ambiguous: scope it, do not add `.first()` to silence it.
|
|
25
|
-
|
|
26
|
-
## Waiting and assertions
|
|
27
|
-
|
|
28
|
-
Actions (`click`, `fill`, `check`) already wait for the element to be attached, visible, stable and enabled. Assertions on locators retry until they pass or time out. Use them; do not sleep.
|
|
29
|
-
|
|
30
|
-
| Need | Use |
|
|
31
|
-
|---|---|
|
|
32
|
-
| Element shown or hidden | `await expect(locator).toBeVisible()` / `.toBeHidden()` |
|
|
33
|
-
| Text | `toHaveText` (full) or `toContainText` |
|
|
34
|
-
| Count | `toHaveCount(n)` |
|
|
35
|
-
| Input value, attribute, focus | `toHaveValue`, `toHaveAttribute`, `toBeFocused` |
|
|
36
|
-
| Navigation | `await expect(page).toHaveURL(/\/dashboard/)` |
|
|
37
|
-
| A request or response | register `page.waitForResponse(...)` **before** the action that triggers it |
|
|
38
|
-
| A non-DOM condition | `await expect.poll(() => readStatus()).toBe('done')` |
|
|
39
|
-
|
|
40
|
-
Wrong and right:
|
|
41
|
-
|
|
42
|
-
```ts
|
|
43
|
-
await page.waitForTimeout(2000); // wrong: too slow or too short
|
|
44
|
-
expect(await page.getByRole('alert').isVisible()).toBe(true); // wrong: checks once, no retry
|
|
45
|
-
expect(await row.textContent()).toBe('Paid'); // wrong: no retry
|
|
46
|
-
|
|
47
|
-
await expect(page.getByRole('alert')).toBeVisible(); // right
|
|
48
|
-
await expect(row).toHaveText('Paid'); // right
|
|
49
|
-
|
|
50
|
-
const saved = page.waitForResponse(r => r.url().includes('/api/profile') && r.ok());
|
|
51
|
-
await page.getByRole('button', { name: 'Save' }).click();
|
|
52
|
-
await saved; // right: registered before the click
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
- `networkidle` is unreliable with websockets, polling or analytics. Wait for the element or URL that proves the page is ready.
|
|
56
|
-
- Every `expect` and every action is awaited. A missing `await` makes a test pass before it checked anything.
|
|
57
|
-
- Do not wrap assertions in try/catch; do not raise timeouts above about 30 s to make something pass.
|
|
58
|
-
|
|
59
|
-
## Flaky tests
|
|
60
|
-
|
|
61
|
-
A flaky test has a cause; retries and longer timeouts only hide it. Classify first:
|
|
62
|
-
|
|
63
|
-
| Symptom | Likely cause | How to confirm |
|
|
64
|
-
|---|---|---|
|
|
65
|
-
| Fails sometimes even alone | Timing: missing await, sleep, race between action and assertion | `npx playwright test <file> --repeat-each=10` (raise the count for rare races) |
|
|
66
|
-
| Passes alone, fails in the suite | Isolation: shared state, shared test data, order dependence | `--grep "<name>" --workers=1` passes, full run fails |
|
|
67
|
-
| Fails only in CI | Environment: viewport, fonts, slower machine, missing service | Compare traces and screenshots from both |
|
|
68
|
-
| Random crashes, "Target closed" | Infrastructure: memory, browser crash | Errors mention the browser, not the app |
|
|
69
|
-
|
|
70
|
-
Fixes: replace sleeps with web-first assertions; register waits before actions; give each test its own data (unique names, per-test setup); avoid module-level mutable variables; set viewport and `reducedMotion: 'reduce'` explicitly when animation or layout matters. For evidence, run with `--trace retain-on-failure --reporter=line --output <tmp>/test-results` and read the error message and the error-context and screenshot files in that output folder. Do not open `show-trace`, `show-report`, `--ui` or `--headed`: they open windows, and nobody is there to look.
|
|
71
|
-
|
|
72
|
-
## Authentication
|
|
73
|
-
|
|
74
|
-
- Use the project's existing auth setup first: a setup project, a global setup that saves `storageState`, a fixture, a test-only login endpoint.
|
|
75
|
-
- Without one, log in once per run through the UI or the API with a **test account the project documents for local development** (seed data, README, fixtures), save `storageState` to the temp folder, and reuse it.
|
|
76
|
-
- Never use real user credentials, never read secrets the task did not give you, and never write auth state files inside the repository.
|
|
77
|
-
- Testing the login page itself: start from an empty state (`storageState: { cookies: [], origins: [] }`).
|
|
78
|
-
- OAuth or SSO with an external provider: do not hit the real provider; check the pages around it or report that the flow could not be exercised.
|
|
79
|
-
|
|
80
|
-
```ts
|
|
81
|
-
// global setup or the first step of a headless check
|
|
82
|
-
await page.goto('/login');
|
|
83
|
-
await page.getByLabel('Email').fill(process.env.E2E_USER ?? 'demo@example.test');
|
|
84
|
-
await page.getByLabel('Password').fill(process.env.E2E_PASSWORD ?? 'from-the-seed-file');
|
|
85
|
-
await page.getByRole('button', { name: 'Sign in' }).click();
|
|
86
|
-
await expect(page).toHaveURL(/dashboard/);
|
|
87
|
-
await page.context().storageState({ path: `${tmpDir}/auth.json` });
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
## Network mocking
|
|
91
|
-
|
|
92
|
-
Mock what you do not own (payment providers, email, maps, analytics) and error states you cannot trigger otherwise. Do not mock the app's own pages, scripts or styles, and prefer the real backend for the happy path.
|
|
93
|
-
|
|
94
|
-
```ts
|
|
95
|
-
await page.route('**/api/orders', route =>
|
|
96
|
-
route.fulfill({ status: 500, json: { error: 'Internal error' } })); // error state
|
|
97
|
-
await page.route('**/api/orders', route => route.fulfill({ json: [] })); // empty state
|
|
98
|
-
await page.route('**/api/orders', async route => { // slow: loading state
|
|
99
|
-
await new Promise(r => setTimeout(r, 3000));
|
|
100
|
-
await route.continue();
|
|
101
|
-
});
|
|
102
|
-
await page.route(/analytics|segment|hotjar/, route => route.abort()); // third-party noise
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
- Register routes before `page.goto`.
|
|
106
|
-
- Every handler calls exactly one of `fulfill`, `continue` or `abort`, otherwise the request hangs.
|
|
107
|
-
- Keep globs narrow (`**/api/orders`, not `**/*`).
|
|
108
|
-
- Loading, empty and error states are where UI changes break most often; mocking is the cheapest way to see them.
|
|
109
|
-
|
|
110
|
-
## Writing new tests (code step only)
|
|
111
|
-
|
|
112
|
-
When the task changes a user flow and the project has an E2E suite:
|
|
113
|
-
|
|
114
|
-
1. **Plan:** list the scenarios from the acceptance criteria: the happy path, the main error state, the empty state, the narrow screen if layout changed. Note what each one must assert.
|
|
115
|
-
2. **Generate:** follow the suite's existing structure, fixtures, page objects and naming. One behaviour per test; several related `expect`s are fine.
|
|
116
|
-
3. **Run** the new tests several times (`--repeat-each=3` at least) and the related existing tests once.
|
|
117
|
-
4. **Repair** failures by finding the cause. If a test fails because the app is wrong, fix the app (in a code step) or report the bug (in a check step).
|
|
118
|
-
|
|
119
|
-
Never make a run green by marking tests `test.skip`, `test.fixme`, `test.fail`, `.only`-ing others, deleting assertions, loosening them (`toHaveText` to `toContainText` without reason), or adding sleeps. A healed test that hides a regression is worse than a red one.
|
|
120
|
-
|
|
121
|
-
## Cypress equivalents
|
|
122
|
-
|
|
123
|
-
| Playwright | Cypress |
|
|
124
|
-
|---|---|
|
|
125
|
-
| `getByRole`, `getByLabel` | `cy.findByRole`, `cy.findByLabelText` if Testing Library is installed; otherwise `cy.contains` and the project's `data-cy` / `data-testid` |
|
|
126
|
-
| web-first `expect(locator)` | `cy.get(...).should(...)` (retries) |
|
|
127
|
-
| `waitForResponse` | `cy.intercept(...).as('x')` then `cy.wait('@x')` |
|
|
128
|
-
| `page.route(...).fulfill` | `cy.intercept(method, url, { statusCode, body })` |
|
|
129
|
-
| `storageState` | `cy.session()` |
|
|
130
|
-
| `npx playwright test <file>` | `npx cypress run --spec <file>` (headless by default; never `cypress open`) |
|
|
131
|
-
|
|
132
|
-
The same bans apply: no `cy.wait(<milliseconds>)`, no skipping failing tests.
|
|
133
|
-
|
|
134
|
-
Adapted from testdino-hq/playwright-skill guides `core/locators.md`, `core/assertions-and-waiting.md`, `core/flaky-tests.md`, `core/authentication.md`, `core/network-mocking.md` and `core/test-agents.md` (MIT, commit 178d1cb537604279d7b54500c4823a6db42b9de6, Copyright (c) 2026 TestDino). Changed: condensed several thousand lines into one page, removed version-specific APIs and vendor mentions, added unattended rules (test accounts, temp folders, no skips), added the Cypress table. The plan, generate, repair loop restates the idea of the Playwright test agents (microsoft/playwright, Apache-2.0) in our own words; no text copied.
|
|
1
|
+
# Playwright patterns: locators, waiting, flakes, auth, mocking
|
|
2
|
+
|
|
3
|
+
Read this when you write, fix or review browser tests or the `steps` of a headless check. In a TAMsense run every browser context, also those of the project's tests, is recorded for the task; scripts of your own name their recordings as [recording.md](recording.md) shows. Examples are Playwright Test in TypeScript; translate to the project's language and runner. If the project uses Cypress, the same ideas apply (see the last section). Check the installed version (`npx playwright --version`) before using an API you are not sure exists in it.
|
|
4
|
+
|
|
5
|
+
## Locators
|
|
6
|
+
|
|
7
|
+
Use the first one that works, in this order:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
page.getByRole('button', { name: 'Save changes' }) // 1. role + accessible name
|
|
11
|
+
page.getByLabel('Email address') // 2. form fields by label
|
|
12
|
+
page.getByText('Welcome back') // 3. non-interactive text
|
|
13
|
+
page.getByPlaceholder('Search') // 4. only if there is no label
|
|
14
|
+
page.getByAltText('Company logo') // 5. images
|
|
15
|
+
page.getByTestId('checkout-summary') // 6. when nothing semantic is stable
|
|
16
|
+
page.locator('.legacy-widget button') // 7. CSS or XPath, last resort
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
- Role locators survive class renames and component swaps, and they fail when the control is not accessible, which is a finding in itself.
|
|
20
|
+
- `exact: true` stops "Log" from matching "Log out".
|
|
21
|
+
- Scope instead of indexing: `page.getByRole('dialog', { name: 'Delete project' }).getByRole('button', { name: 'Delete' })` beats `.nth(2)`.
|
|
22
|
+
- Rows and cards: `page.getByRole('row').filter({ hasText: 'Invoice 42' }).getByRole('button', { name: 'Open' })`.
|
|
23
|
+
- Use the project's existing `data-testid` convention if it has one; do not add test ids to product code during a check step.
|
|
24
|
+
- Strict mode errors ("resolved to 2 elements") mean the locator is ambiguous: scope it, do not add `.first()` to silence it.
|
|
25
|
+
|
|
26
|
+
## Waiting and assertions
|
|
27
|
+
|
|
28
|
+
Actions (`click`, `fill`, `check`) already wait for the element to be attached, visible, stable and enabled. Assertions on locators retry until they pass or time out. Use them; do not sleep.
|
|
29
|
+
|
|
30
|
+
| Need | Use |
|
|
31
|
+
|---|---|
|
|
32
|
+
| Element shown or hidden | `await expect(locator).toBeVisible()` / `.toBeHidden()` |
|
|
33
|
+
| Text | `toHaveText` (full) or `toContainText` |
|
|
34
|
+
| Count | `toHaveCount(n)` |
|
|
35
|
+
| Input value, attribute, focus | `toHaveValue`, `toHaveAttribute`, `toBeFocused` |
|
|
36
|
+
| Navigation | `await expect(page).toHaveURL(/\/dashboard/)` |
|
|
37
|
+
| A request or response | register `page.waitForResponse(...)` **before** the action that triggers it |
|
|
38
|
+
| A non-DOM condition | `await expect.poll(() => readStatus()).toBe('done')` |
|
|
39
|
+
|
|
40
|
+
Wrong and right:
|
|
41
|
+
|
|
42
|
+
```ts
|
|
43
|
+
await page.waitForTimeout(2000); // wrong: too slow or too short
|
|
44
|
+
expect(await page.getByRole('alert').isVisible()).toBe(true); // wrong: checks once, no retry
|
|
45
|
+
expect(await row.textContent()).toBe('Paid'); // wrong: no retry
|
|
46
|
+
|
|
47
|
+
await expect(page.getByRole('alert')).toBeVisible(); // right
|
|
48
|
+
await expect(row).toHaveText('Paid'); // right
|
|
49
|
+
|
|
50
|
+
const saved = page.waitForResponse(r => r.url().includes('/api/profile') && r.ok());
|
|
51
|
+
await page.getByRole('button', { name: 'Save' }).click();
|
|
52
|
+
await saved; // right: registered before the click
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
- `networkidle` is unreliable with websockets, polling or analytics. Wait for the element or URL that proves the page is ready.
|
|
56
|
+
- Every `expect` and every action is awaited. A missing `await` makes a test pass before it checked anything.
|
|
57
|
+
- Do not wrap assertions in try/catch; do not raise timeouts above about 30 s to make something pass.
|
|
58
|
+
|
|
59
|
+
## Flaky tests
|
|
60
|
+
|
|
61
|
+
A flaky test has a cause; retries and longer timeouts only hide it. Classify first:
|
|
62
|
+
|
|
63
|
+
| Symptom | Likely cause | How to confirm |
|
|
64
|
+
|---|---|---|
|
|
65
|
+
| Fails sometimes even alone | Timing: missing await, sleep, race between action and assertion | `npx playwright test <file> --repeat-each=10` (raise the count for rare races) |
|
|
66
|
+
| Passes alone, fails in the suite | Isolation: shared state, shared test data, order dependence | `--grep "<name>" --workers=1` passes, full run fails |
|
|
67
|
+
| Fails only in CI | Environment: viewport, fonts, slower machine, missing service | Compare traces and screenshots from both |
|
|
68
|
+
| Random crashes, "Target closed" | Infrastructure: memory, browser crash | Errors mention the browser, not the app |
|
|
69
|
+
|
|
70
|
+
Fixes: replace sleeps with web-first assertions; register waits before actions; give each test its own data (unique names, per-test setup); avoid module-level mutable variables; set viewport and `reducedMotion: 'reduce'` explicitly when animation or layout matters. For evidence, run with `--trace retain-on-failure --reporter=line --output <tmp>/test-results` and read the error message and the error-context and screenshot files in that output folder. Do not open `show-trace`, `show-report`, `--ui` or `--headed`: they open windows, and nobody is there to look.
|
|
71
|
+
|
|
72
|
+
## Authentication
|
|
73
|
+
|
|
74
|
+
- Use the project's existing auth setup first: a setup project, a global setup that saves `storageState`, a fixture, a test-only login endpoint.
|
|
75
|
+
- Without one, log in once per run through the UI or the API with a **test account the project documents for local development** (seed data, README, fixtures), save `storageState` to the temp folder, and reuse it.
|
|
76
|
+
- Never use real user credentials, never read secrets the task did not give you, and never write auth state files inside the repository.
|
|
77
|
+
- Testing the login page itself: start from an empty state (`storageState: { cookies: [], origins: [] }`).
|
|
78
|
+
- OAuth or SSO with an external provider: do not hit the real provider; check the pages around it or report that the flow could not be exercised.
|
|
79
|
+
|
|
80
|
+
```ts
|
|
81
|
+
// global setup or the first step of a headless check
|
|
82
|
+
await page.goto('/login');
|
|
83
|
+
await page.getByLabel('Email').fill(process.env.E2E_USER ?? 'demo@example.test');
|
|
84
|
+
await page.getByLabel('Password').fill(process.env.E2E_PASSWORD ?? 'from-the-seed-file');
|
|
85
|
+
await page.getByRole('button', { name: 'Sign in' }).click();
|
|
86
|
+
await expect(page).toHaveURL(/dashboard/);
|
|
87
|
+
await page.context().storageState({ path: `${tmpDir}/auth.json` });
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Network mocking
|
|
91
|
+
|
|
92
|
+
Mock what you do not own (payment providers, email, maps, analytics) and error states you cannot trigger otherwise. Do not mock the app's own pages, scripts or styles, and prefer the real backend for the happy path.
|
|
93
|
+
|
|
94
|
+
```ts
|
|
95
|
+
await page.route('**/api/orders', route =>
|
|
96
|
+
route.fulfill({ status: 500, json: { error: 'Internal error' } })); // error state
|
|
97
|
+
await page.route('**/api/orders', route => route.fulfill({ json: [] })); // empty state
|
|
98
|
+
await page.route('**/api/orders', async route => { // slow: loading state
|
|
99
|
+
await new Promise(r => setTimeout(r, 3000));
|
|
100
|
+
await route.continue();
|
|
101
|
+
});
|
|
102
|
+
await page.route(/analytics|segment|hotjar/, route => route.abort()); // third-party noise
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
- Register routes before `page.goto`.
|
|
106
|
+
- Every handler calls exactly one of `fulfill`, `continue` or `abort`, otherwise the request hangs.
|
|
107
|
+
- Keep globs narrow (`**/api/orders`, not `**/*`).
|
|
108
|
+
- Loading, empty and error states are where UI changes break most often; mocking is the cheapest way to see them.
|
|
109
|
+
|
|
110
|
+
## Writing new tests (code step only)
|
|
111
|
+
|
|
112
|
+
When the task changes a user flow and the project has an E2E suite:
|
|
113
|
+
|
|
114
|
+
1. **Plan:** list the scenarios from the acceptance criteria: the happy path, the main error state, the empty state, the narrow screen if layout changed. Note what each one must assert.
|
|
115
|
+
2. **Generate:** follow the suite's existing structure, fixtures, page objects and naming. One behaviour per test; several related `expect`s are fine.
|
|
116
|
+
3. **Run** the new tests several times (`--repeat-each=3` at least) and the related existing tests once.
|
|
117
|
+
4. **Repair** failures by finding the cause. If a test fails because the app is wrong, fix the app (in a code step) or report the bug (in a check step).
|
|
118
|
+
|
|
119
|
+
Never make a run green by marking tests `test.skip`, `test.fixme`, `test.fail`, `.only`-ing others, deleting assertions, loosening them (`toHaveText` to `toContainText` without reason), or adding sleeps. A healed test that hides a regression is worse than a red one.
|
|
120
|
+
|
|
121
|
+
## Cypress equivalents
|
|
122
|
+
|
|
123
|
+
| Playwright | Cypress |
|
|
124
|
+
|---|---|
|
|
125
|
+
| `getByRole`, `getByLabel` | `cy.findByRole`, `cy.findByLabelText` if Testing Library is installed; otherwise `cy.contains` and the project's `data-cy` / `data-testid` |
|
|
126
|
+
| web-first `expect(locator)` | `cy.get(...).should(...)` (retries) |
|
|
127
|
+
| `waitForResponse` | `cy.intercept(...).as('x')` then `cy.wait('@x')` |
|
|
128
|
+
| `page.route(...).fulfill` | `cy.intercept(method, url, { statusCode, body })` |
|
|
129
|
+
| `storageState` | `cy.session()` |
|
|
130
|
+
| `npx playwright test <file>` | `npx cypress run --spec <file>` (headless by default; never `cypress open`) |
|
|
131
|
+
|
|
132
|
+
The same bans apply: no `cy.wait(<milliseconds>)`, no skipping failing tests.
|
|
133
|
+
|
|
134
|
+
Adapted from testdino-hq/playwright-skill guides `core/locators.md`, `core/assertions-and-waiting.md`, `core/flaky-tests.md`, `core/authentication.md`, `core/network-mocking.md` and `core/test-agents.md` (MIT, commit 178d1cb537604279d7b54500c4823a6db42b9de6, Copyright (c) 2026 TestDino). Changed: condensed several thousand lines into one page, removed version-specific APIs and vendor mentions, added unattended rules (test accounts, temp folders, no skips), added the Cypress table. The plan, generate, repair loop restates the idea of the Playwright test agents (microsoft/playwright, Apache-2.0) in our own words; no text copied.
|
|
@@ -1,58 +1,61 @@
|
|
|
1
|
-
# Recording the browser for the task
|
|
2
|
-
|
|
3
|
-
Read this before any script that opens a browser during a TAMsense run. People on the task watch what your browser did: every page is recorded as a replay (the DOM, clicks and typing, played back in the TAMsense player) and attached to the task when the run ends.
|
|
4
|
-
|
|
5
|
-
## What happens by itself
|
|
6
|
-
|
|
7
|
-
- When the `TAMSENSE_RECORDER` environment variable is set, every Node process you start loads the TAMsense recorder, and every Playwright browser context is recorded, also the ones opened by the project's own E2E tests. You cannot switch this off, and you do not need to.
|
|
8
|
-
- Requests from the tested site to the TAMsense tracker (`/api/events` with a `?key=` or sent to the TAMsense server) are answered by the recorder and never leave the browser, so your runs do not reach the product's analytics.
|
|
9
|
-
- Requests other than GET, HEAD and OPTIONS to an address outside this computer (anything but `localhost`, `*.localhost` and loopback addresses) are aborted, so a test run never writes to an outside service. The summary from `closeRecorded` counts them in `blockedExternalWrites`. If a flow you check needs such a write, it was set up against an outside service: say so instead of working around it.
|
|
10
|
-
- Password fields are masked in the replay.
|
|
11
|
-
- Recordings you did not attach yourself are attached when the run ends. Unnamed ones get the step name as their title, so name yours.
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
await
|
|
33
|
-
await page
|
|
34
|
-
await
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
- `
|
|
43
|
-
- `
|
|
44
|
-
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
1
|
+
# Recording the browser for the task
|
|
2
|
+
|
|
3
|
+
Read this before any script that opens a browser during a TAMsense run. People on the task watch what your browser did: every page is recorded as a replay (the DOM, clicks and typing, played back in the TAMsense player) and attached to the task when the run ends.
|
|
4
|
+
|
|
5
|
+
## What happens by itself
|
|
6
|
+
|
|
7
|
+
- When the `TAMSENSE_RECORDER` environment variable is set, every Node process you start loads the TAMsense recorder, and every Playwright browser context is recorded, also the ones opened by the project's own E2E tests. You cannot switch this off, and you do not need to.
|
|
8
|
+
- Requests from the tested site to the TAMsense tracker (`/api/events` with a `?key=` or sent to the TAMsense server) are answered by the recorder and never leave the browser, so your runs do not reach the product's analytics.
|
|
9
|
+
- Requests other than GET, HEAD and OPTIONS to an address outside this computer (anything but `localhost`, `*.localhost` and loopback addresses) are aborted, so a test run never writes to an outside service. The summary from `closeRecorded` counts them in `blockedExternalWrites`. If a flow you check needs such a write, it was set up against an outside service: say so instead of working around it.
|
|
10
|
+
- Password fields are masked in the replay.
|
|
11
|
+
- Recordings you did not attach yourself are attached when the run ends. Unnamed ones get the step name as their title, so name yours.
|
|
12
|
+
- The task shows who made each recording: your role, the step, the round and the computer. You do not write that yourself.
|
|
13
|
+
|
|
14
|
+
## What you add in every browser script
|
|
15
|
+
|
|
16
|
+
Load the recorder and open each context through it, with a title and a purpose:
|
|
17
|
+
|
|
18
|
+
```js
|
|
19
|
+
import { createRequire } from 'node:module'
|
|
20
|
+
|
|
21
|
+
const recorder = process.env.TAMSENSE_RECORDER ? createRequire(import.meta.url)(process.env.TAMSENSE_RECORDER) : null
|
|
22
|
+
|
|
23
|
+
const context = recorder
|
|
24
|
+
? await recorder.newRecordedContext(browser, {
|
|
25
|
+
title: 'Checkout testing, attempt 2',
|
|
26
|
+
purpose: 'Paying with a saved card after the fix to the total',
|
|
27
|
+
context: { viewport: { width: 1280, height: 900 } },
|
|
28
|
+
})
|
|
29
|
+
: await browser.newContext({ viewport: { width: 1280, height: 900 } })
|
|
30
|
+
const page = await context.newPage()
|
|
31
|
+
|
|
32
|
+
await page.goto(url)
|
|
33
|
+
await recorder?.mark(page, 'Opened checkout')
|
|
34
|
+
await page.getByRole('button', { name: 'Pay' }).click()
|
|
35
|
+
await recorder?.mark(page, 'Paid, the receipt is shown')
|
|
36
|
+
|
|
37
|
+
const notes = 'Opened checkout with a saved card and clicked Pay. Expected the receipt with the corrected total of $42: it showed $42. Passed.'
|
|
38
|
+
const recorded = recorder ? await recorder.closeRecorded(context, { notes }) : await context.close()
|
|
39
|
+
console.log(JSON.stringify(recorded, null, 2))
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
- `title`: what it shows, at most 60 characters. Number design variants ("Design option 1", "Design option 2") and name test runs by their goal and attempt ("Feature testing", "Second attempt at feature testing"). Never generic names like "screenshot1" or "test".
|
|
43
|
+
- `purpose`: one line, at most 160 characters, on why it was made ("Comparing two layouts for the empty orders state").
|
|
44
|
+
- `mark(page, label)`: call it at each meaningful step (opened a screen, submitted a form, saw the error). Marks show on the replay's timeline, and a click on one jumps there.
|
|
45
|
+
- `closeRecorded(context, { notes })`: closes the context, finishes the files and returns where they are. Close every context this way, or at least with `context.close()`, before the script ends.
|
|
46
|
+
- `notes`: what you tested in this browser and what happened, in your own words, at most 600 characters: what you opened, what you clicked or typed, what you expected, and whether it passed or failed. People see them on the replay card and above the player before they press play, so write them for someone who did not read your report. Write them after the steps ran, from what you actually saw. A recording without notes shows the marks you set instead, and without marks, "No notes from the agent".
|
|
47
|
+
- Without `TAMSENSE_RECORDER` (not a TAMsense run) the script works as before.
|
|
48
|
+
|
|
49
|
+
## When to add a video
|
|
50
|
+
|
|
51
|
+
A replay rebuilds the page from its DOM, so it cannot show pixels drawn by the page itself. Pass `video: true` to `newRecordedContext` as well when the screen has:
|
|
52
|
+
|
|
53
|
+
- a `<canvas>` (charts drawn on canvas, editors, games), a map, or a `<video>` player;
|
|
54
|
+
- iframes from another site (payment forms, embedded widgets);
|
|
55
|
+
- or when someone in the chat asks for a video.
|
|
56
|
+
|
|
57
|
+
`closeRecorded` warns when it saw a canvas, a video or a cross-origin iframe on a page recorded without video. Videos are larger (up to 200 MB each, 5 per run), so keep them short.
|
|
58
|
+
|
|
59
|
+
## Attaching earlier
|
|
60
|
+
|
|
61
|
+
Everything is attached at the end of the run. To attach a recording sooner, or with a better title, call `attach_replay` (the `.rrweb.ndjson` file) or `attach_video` (the `.webm` file) with the path from `closeRecorded`, a title, a purpose and notes (the notes you gave `closeRecorded` are used when you leave them out). A run attaches at most 20 replays and 5 videos; the ones you named come first.
|