openspec-playwright 0.3.45 → 0.3.46
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/templates/e2e-command.md +159 -1019
package/templates/e2e-command.md
CHANGED
|
@@ -11,576 +11,165 @@
|
|
|
11
11
|
- **Page Objects** (all mode): `tests/playwright/pages/<Route>Page.ts`
|
|
12
12
|
- **Auth setup**: `tests/playwright/auth.setup.ts` (if auth required)
|
|
13
13
|
- **Report**: `openspec/reports/playwright-e2e-<name>-<timestamp>.md`
|
|
14
|
-
- **App Bug Registry**: `openspec/reports/app-bug-registry.md` (cumulative
|
|
14
|
+
- **App Bug Registry**: `openspec/reports/app-bug-registry.md` (cumulative)
|
|
15
15
|
- **Test plan**: `openspec/changes/<name>/specs/playwright/test-plan.md` (change mode only)
|
|
16
16
|
|
|
17
17
|
## Architecture
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
| Mode | Command | Route source | Output |
|
|
20
|
+
| ------ | ------------------- | ------------------------ | ------------------------------- |
|
|
21
|
+
| Change | `/opsx:e2e <name>` | OpenSpec specs | `changes/<name>/<name>.spec.ts` |
|
|
22
|
+
| All | `/opsx:e2e all` | sitemap + homepage crawl | `pages/*.ts` (Page Objects) |
|
|
20
23
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
| Change | `/opsx:e2e <name>` | OpenSpec specs | `changes/<name>/<name>.spec.ts` |
|
|
24
|
-
| All | `/opsx:e2e all` | sitemap + homepage crawl | `pages/*.ts` (Page Objects) |
|
|
24
|
+
> **Full regression is opt-in only.** `openspec-pw run <name>` → one spec file. Do NOT run `npx playwright test` (no file) or `--only-changed` unless user explicitly requests.
|
|
25
|
+
> **Roles**: Planner (Steps 4–5) → test-plan.md; Generator (Step 6) → `.spec.ts` + Page Objects; Healer (Step 9) → repairs failures via MCP.
|
|
25
26
|
|
|
26
|
-
|
|
27
|
+
Browser exploration is tool-agnostic: gstack (`/browse`), Playwright MCP, or `openspec-pw explore --parallel N`.
|
|
27
28
|
|
|
28
|
-
|
|
29
|
+
## Testing Principles
|
|
29
30
|
|
|
30
|
-
|
|
31
|
+
**UI first.** Every assertion about visible UI state must use `page.getByRole/ByLabel/ByText + expect()`. `page.request` is acceptable only for precondition setup or HTTP-level mocking via `page.route()`.
|
|
31
32
|
|
|
32
|
-
**Browser exploration is tool-agnostic**: gstack (`/browse`), Playwright MCP (`browser_navigate` / `browser_snapshot` / etc.), and `openspec-pw explore --parallel N` all work — pick whichever is installed. Playwright MCP is always required (Healer uses it), so reuse it for exploration if you don't have gstack.
|
|
33
|
-
|
|
34
|
-
## Setup / Teardown
|
|
35
|
-
|
|
36
|
-
Playwright supports two approaches for global lifecycle hooks. **openspec-playwright uses project dependencies** (the recommended approach) for full feature support.
|
|
37
|
-
|
|
38
|
-
### Comparison
|
|
39
|
-
|
|
40
|
-
| Feature | Project Dependencies | globalSetup/globalTeardown |
|
|
41
|
-
|---------|---------------------|---------------------------|
|
|
42
|
-
| HTML report visibility | ✅ Shown as project | ❌ Not shown |
|
|
43
|
-
| Trace recording | ✅ Full support | ❌ Not supported |
|
|
44
|
-
| Playwright fixtures | ✅ Fully supported | ❌ Not supported |
|
|
45
|
-
| Browser via fixture | ✅ Automatic | ❌ Manual launch |
|
|
46
|
-
|
|
47
|
-
### Current Implementation
|
|
48
|
-
|
|
49
|
-
**Setup project** (enabled by default):
|
|
50
|
-
- `tests/playwright/auth.setup.ts` — authenticates once, saves session to `./playwright/.auth/user.json`
|
|
51
|
-
- All test projects depend on setup via `dependencies: ['setup']`
|
|
52
|
-
|
|
53
|
-
**Teardown project** (optional, disabled by default):
|
|
54
|
-
- `tests/playwright/global.teardown.ts` — runs AFTER all tests complete
|
|
55
|
-
- Use for: database cleanup, uploaded file removal, cache invalidation
|
|
56
|
-
- Enable by uncommenting in `playwright.config.ts`
|
|
57
|
-
|
|
58
|
-
### When to Enable Teardown
|
|
59
|
-
|
|
60
|
-
Enable teardown when your tests create persistent data that should be cleaned up:
|
|
61
|
-
|
|
62
|
-
| Scenario | Action |
|
|
63
|
-
|----------|--------|
|
|
64
|
-
| Tests create database records | ✅ Enable teardown, add DB cleanup |
|
|
65
|
-
| Tests upload files | ✅ Enable teardown, add file cleanup |
|
|
66
|
-
| Tests only read data | ❌ No teardown needed |
|
|
67
|
-
| Tests use ephemeral/isolated environments | ❌ No teardown needed |
|
|
68
|
-
|
|
69
|
-
### Enabling Teardown
|
|
70
|
-
|
|
71
|
-
1. Copy template: `cp templates/global.teardown.ts tests/playwright/global.teardown.ts`
|
|
72
|
-
2. Customize cleanup logic in the file
|
|
73
|
-
3. Uncomment in `playwright.config.ts`:
|
|
74
|
-
```typescript
|
|
75
|
-
projects: [
|
|
76
|
-
{ name: 'setup', testMatch: /.*\.setup\.ts/ },
|
|
77
|
-
{ name: 'teardown', testMatch: /global\.teardown\.ts/ }, // Uncomment
|
|
78
|
-
{
|
|
79
|
-
name: 'chromium',
|
|
80
|
-
// ...
|
|
81
|
-
teardown: 'teardown', // Uncomment
|
|
82
|
-
},
|
|
83
|
-
],
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
## Testing principles
|
|
87
|
-
|
|
88
|
-
**UI first** — Test every user flow through the browser UI. E2E validates that users can accomplish tasks in the real interface, not just that the backend responds correctly.
|
|
89
|
-
|
|
90
|
-
```
|
|
91
|
-
用户操作 → 浏览器 UI → 后端 → 数据库 → UI 反馈
|
|
92
33
|
```
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
**Decision rule (per assertion)**:
|
|
97
|
-
|
|
98
|
-
```
|
|
99
|
-
Can the user SEE this on screen?
|
|
100
|
-
→ Yes → MUST use: page.getByRole/ByLabel/ByText + expect()
|
|
34
|
+
Can user SEE this on screen?
|
|
35
|
+
→ Yes → MUST use: UI selector + expect()
|
|
101
36
|
→ No → Record reason → page.request acceptable
|
|
102
37
|
```
|
|
103
38
|
|
|
104
|
-
**Business logic assertion rule
|
|
105
|
-
|
|
106
|
-
UI assertions verify **rendering** correctness, not **calculation** correctness. A UI that correctly displays a wrong value will pass UI-only tests.
|
|
107
|
-
|
|
108
|
-
```
|
|
109
|
-
Is the assertion about a computed/counted/calculated value?
|
|
110
|
-
(e.g., balance, total, discount, count, percentage, score, ranking)
|
|
111
|
-
→ Yes → Use page.request to fetch backend data → assert the raw value
|
|
112
|
-
→ No → UI assertion is sufficient
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
**Mock data rule:**
|
|
116
|
-
|
|
117
|
-
- **Frontend: forbidden.** UI interactions must use real browser + real app data. Mocking frontend code (JS variables, component state, module-level stubs) hides real integration issues. If the frontend cannot reach a scenario through normal UI flow → **ask the user**.
|
|
118
|
-
- **API: allowed at HTTP level.** Use `page.route()` to intercept and mock API responses (status codes, body data, latency) when:
|
|
119
|
-
- Triggering HTTP 5xx/4xx error responses (hard to reach via UI)
|
|
120
|
-
- Edge cases requiring pre-condition data that UI cannot set up
|
|
121
|
-
- Third-party API failures (payment, SMS, email providers)
|
|
122
|
-
- **Scope: API level only** — `page.route()` intercepts HTTP; do NOT mock at the database or backend service level. Mocking below the HTTP layer bypasses the real API contract and produces false confidence.
|
|
123
|
-
- **User consent required.** Before using `page.route()` mocking, stop and ask:
|
|
124
|
-
```
|
|
125
|
-
API mocking needed for: <reason>
|
|
126
|
-
Mocked endpoint: <URL or pattern>
|
|
127
|
-
Expected behavior: <what the test verifies>
|
|
128
|
-
Reply **yes** to proceed, or tell me to find a UI-based approach instead.
|
|
129
|
-
```
|
|
130
|
-
If the user says no → attempt a UI-based approach or skip that test case.
|
|
131
|
-
|
|
132
|
-
**Examples where API assertion is required:
|
|
133
|
-
|
|
134
|
-
```typescript
|
|
135
|
-
// ❌ UI-only assertion — hides calculation bugs
|
|
136
|
-
await page.getByText('¥800').click(); // buy item
|
|
137
|
-
await expect(page.getByText('总金额: ¥800')).toBeVisible(); // passes even if backend rounded wrong
|
|
39
|
+
**Business logic assertion rule**: computed/counted values (balance, total, count, percentage) MUST use API assertion. UI assertions verify rendering, not calculation.
|
|
138
40
|
|
|
139
|
-
|
|
140
|
-
const order = await page.request.get(`${BASE_URL}/api/orders/${orderId}`);
|
|
141
|
-
const body = await order.json();
|
|
142
|
-
expect(body.total).toBe(800); // backend calculation is verified
|
|
41
|
+
**Mock data rule**: Frontend mocking forbidden (no JS stubs, module stubs). API mocking via `page.route()` allowed for 4xx/5xx, edge cases, third-party failures — with user consent and at HTTP level only. Never mock below HTTP (DB, backend service).
|
|
143
42
|
|
|
144
|
-
|
|
145
|
-
await page.getByRole('button', { name: '点赞' }).click();
|
|
146
|
-
await expect(page.getByText('1 个赞')).toBeVisible(); // passes even if POST failed silently
|
|
43
|
+
**API assertion examples**: computed/counted values → MUST use `page.request` to verify, not UI:
|
|
147
44
|
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
const
|
|
151
|
-
expect(
|
|
45
|
+
```
|
|
46
|
+
// ✅ API assertion catches calculation bugs
|
|
47
|
+
const order = await page.request.get(`${BASE_URL}/api/orders/${id}`);
|
|
48
|
+
expect((await order.json()).total).toBe(800);
|
|
152
49
|
|
|
153
50
|
// ✅ Optimistic update with API verification
|
|
154
|
-
await page.
|
|
155
|
-
await
|
|
156
|
-
await page.waitForResponse(r => r.url().includes('/api/like')); // wait for backend
|
|
157
|
-
const post = await page.request.get(`${BASE_URL}/api/posts/${postId}`);
|
|
158
|
-
expect((await post.json()).likeCount).toBeGreaterThan(0); // verify persistence
|
|
51
|
+
await expect(page.getByText('总金额: ¥800')).toBeVisible();
|
|
52
|
+
await page.waitForResponse(r => r.url().includes('/api/like'));
|
|
159
53
|
```
|
|
160
54
|
|
|
161
|
-
**Never use API
|
|
55
|
+
**Never use API to replace routine UI flow.** If a test completes in <200ms, it skips real UI.
|
|
162
56
|
|
|
163
57
|
## Steps
|
|
164
58
|
|
|
165
59
|
### 1. Select the change or mode
|
|
166
60
|
|
|
167
|
-
**Change mode
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
**"all" mode** (`/opsx:e2e all` — no OpenSpec needed):
|
|
175
|
-
|
|
176
|
-
- Announce: "Mode: full app exploration + Page Object discovery"
|
|
177
|
-
- **Goal**: Discover new routes, extract selectors, and build `pages/*.ts` Page Objects — accumulated asset for future Change tests
|
|
178
|
-
- **Route discovery** (in order):
|
|
179
|
-
1. **sitemap.xml**: navigate to `${BASE_URL}/sitemap.xml` → parse URLs
|
|
180
|
-
2. **Link extraction**: Navigate to `${BASE_URL}/` → evaluate JS to extract all `<a href>`:
|
|
181
|
-
```javascript
|
|
182
|
-
// Extract all internal links from current page
|
|
183
|
-
() => {
|
|
184
|
-
const origin = window.location.origin;
|
|
185
|
-
const links = Array.from(document.querySelectorAll('a[href]'));
|
|
186
|
-
return links
|
|
187
|
-
.map(a => a.href)
|
|
188
|
-
.filter(h => h.startsWith(origin) && !h.includes('/logout') && !h.includes('/api/'))
|
|
189
|
-
.map(h => new URL(h).pathname);
|
|
190
|
-
}
|
|
191
|
-
```
|
|
192
|
-
3. **Fallback common paths**: `/`, `/login`, `/dashboard`, `/admin`, `/profile`, `/settings`
|
|
193
|
-
|
|
194
|
-
**Decision table — route discovery fallback:**
|
|
61
|
+
**Change mode**: Use provided name, infer from context, or auto-select if only one exists. If ambiguous → `openspec list --json` + AskUserQuestion. If specs empty → STOP, suggest "all" mode.
|
|
62
|
+
|
|
63
|
+
**"all" mode**: Route discovery priority:
|
|
64
|
+
1. sitemap.xml (navigate to `${BASE_URL}/sitemap.xml` → parse URLs)
|
|
65
|
+
2. Link extraction (navigate home → extract `<a href>` for internal paths)
|
|
66
|
+
3. Fallback common paths (`/`, `/login`, `/dashboard`, `/admin`, `/profile`, `/settings`)
|
|
195
67
|
|
|
196
68
|
| Situation | Action |
|
|
197
|
-
|
|
|
198
|
-
|
|
|
199
|
-
|
|
|
200
|
-
| Link extraction
|
|
201
|
-
|
|
|
202
|
-
| Duplicate routes from multiple sources | Deduplicate by pathname |
|
|
69
|
+
| --------- | ------ |
|
|
70
|
+
| sitemap.xml returns 200 with URLs | Parse → extract pathname |
|
|
71
|
+
| sitemap.xml 404/5xx | Skip → link extraction |
|
|
72
|
+
| Link extraction → 0 links | Fallback common paths |
|
|
73
|
+
| Duplicates | Deduplicate by pathname |
|
|
203
74
|
|
|
204
|
-
|
|
205
|
-
- Group routes: Guest vs Protected (by attempting direct access)
|
|
75
|
+
Persist routes to `app-knowledge.md` → **Routes** table (replace entire table, do not append). Group routes: Guest vs Protected (by direct access attempt).
|
|
206
76
|
|
|
207
77
|
### 2. Detect auth
|
|
208
78
|
|
|
209
|
-
**Change mode**: Read specs
|
|
210
|
-
|
|
211
|
-
**"all" mode**: Detect auth by attempting to access known protected paths (e.g. `/dashboard`, `/profile`). If redirected to `/login` → auth required.
|
|
212
|
-
|
|
213
|
-
**Auth detection — both modes** (BOTH conditions required):
|
|
214
|
-
|
|
215
|
-
**Condition A — Explicit markers**: "login", "signin", "logout", "authenticate", "protected", "authenticated", "session", "unauthorized", "jwt", "token", "refresh", "middleware"
|
|
79
|
+
**Change mode**: Read specs, detect auth from keywords. **"all" mode**: Try accessing protected paths → redirected to `/login` → auth required.
|
|
216
80
|
|
|
217
|
-
**
|
|
218
|
-
|
|
219
|
-
**
|
|
220
|
-
|
|
221
|
-
**Confidence — decision table:**
|
|
81
|
+
**Both conditions required**:
|
|
82
|
+
- **A — Explicit markers**: "login", "signin", "logout", "authenticate", "protected", "session", "unauthorized", "jwt", "token", "refresh", "middleware"
|
|
83
|
+
- **B — Context indicators**: Protected routes, role mentions ("admin", "user"), redirect flows
|
|
222
84
|
|
|
223
85
|
| Confidence | Condition | Action |
|
|
224
|
-
|
|
|
225
|
-
| High | Multiple markers
|
|
226
|
-
| Medium | Single marker, context unclear | Proceed + note
|
|
227
|
-
| Low | No markers
|
|
86
|
+
| ---------- | --------- | ------ |
|
|
87
|
+
| High | Multiple markers + context indicators | Auto-proceed |
|
|
88
|
+
| Medium | Single marker, context unclear | Proceed + note |
|
|
89
|
+
| Low | No markers | Skip auth, test as guest |
|
|
228
90
|
|
|
229
|
-
|
|
91
|
+
Exclude false positives: HTTP header examples and code snippets do not count.
|
|
230
92
|
|
|
231
|
-
|
|
93
|
+
### 3. Validate environment
|
|
232
94
|
|
|
233
95
|
```bash
|
|
234
96
|
npx playwright test tests/playwright/seed.spec.ts --project=chromium
|
|
235
97
|
```
|
|
236
98
|
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
**If seed test fails**: Stop and report. Fix the environment before proceeding.
|
|
99
|
+
If seed test fails → STOP. Fix environment before proceeding. This validates BASE_URL, auth setup, and Playwright are functional.
|
|
240
100
|
|
|
241
101
|
### 4. Explore application
|
|
242
102
|
|
|
243
|
-
**Prerequisites**:
|
|
244
|
-
1. At least one browser exploration tool installed (gstack / Playwright MCP / `openspec-pw explore`).
|
|
245
|
-
2. seed test pass
|
|
246
|
-
3. BASE_URL must be verified reachable (see 4.1)
|
|
247
|
-
|
|
248
|
-
If auth is required and `auth.setup.ts` already exists → auth is ready. If auth is not yet configured → use the workaround below (Option B for protected routes).
|
|
249
|
-
|
|
250
|
-
#### 4.1. Verify BASE_URL + Read app-knowledge.md
|
|
251
|
-
|
|
252
|
-
1. **Verify BASE_URL**: navigate to `<BASE_URL>` → if HTTP 5xx → **STOP: backend error. Fix app first.**
|
|
253
|
-
2. **Read app-knowledge.md**: known risks, project conventions
|
|
254
|
-
3. **Routes** (from Step 1): use already-discovered routes — no need to re-extract
|
|
255
|
-
|
|
256
|
-
#### 4.2. Explore each route
|
|
257
|
-
|
|
258
|
-
For each route: navigate → check console for errors → snapshot DOM → screenshot.
|
|
259
|
-
|
|
260
|
-
#### Alternative: parallel exploration (via `openspec-pw explore`)
|
|
261
|
-
|
|
262
|
-
If you have ≥5 routes, skip 4.2 and use the dedicated CLI for genuine parallel exploration:
|
|
263
|
-
|
|
264
|
-
```bash
|
|
265
|
-
openspec-pw explore --parallel 4 # 4 independent Chromium workers
|
|
266
|
-
openspec-pw explore --dry-run # preview chunk assignment first
|
|
267
|
-
```
|
|
268
|
-
|
|
269
|
-
**Why**: a single shared browser instance with one active page causes `Promise.allSettled` on navigation/snapshot commands to serialize execution and create navigation state conflicts. `openspec-pw explore` launches N independent browser processes, each with its own Chromium context, for genuine parallelism.
|
|
103
|
+
**Prerequisites**: browser tool installed, seed test passed, BASE_URL reachable.
|
|
270
104
|
|
|
271
|
-
**
|
|
105
|
+
**4.1. Verify BASE_URL**: navigate → if HTTP 5xx → **STOP: backend error**. Read `app-knowledge.md` for known risks and conventions.
|
|
272
106
|
|
|
273
|
-
**
|
|
274
|
-
|
|
275
|
-
| Signal | Meaning | Action |
|
|
276
|
-
| ----------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------- |
|
|
277
|
-
| HTTP 5xx or unreachable | Backend/server error | **STOP** — tell user: "App has a backend error (HTTP <code>). Fix it, then re-run `/opsx:e2e <name>` to re-explore." |
|
|
278
|
-
| JS error in console | App runtime error | **STOP** — tell user: "Page has JS errors. Fix them, then re-run `/opsx:e2e <name>` to re-explore." |
|
|
279
|
-
| HTTP 404 | Route not in app (metadata issue) | Continue — mark `⚠️ route not found` in app-exploration.md |
|
|
280
|
-
| Auth required, no credentials | Missing auth setup | Continue — skip protected routes, explore login page |
|
|
281
|
-
| Suspicious network request | API returned 4xx/5xx | Continue — mark `⚠️ API error: <endpoint> returned <code>` in app-exploration.md |
|
|
282
|
-
|
|
283
|
-
**Redirect / Refresh loop detection** — run after initial navigate:
|
|
284
|
-
|
|
285
|
-
```
|
|
286
|
-
// 1. Initial capture
|
|
287
|
-
navigate <url>
|
|
288
|
-
wait for network-idle // wait for SPA hydration
|
|
289
|
-
evaluate "window.location.href" // url1
|
|
290
|
-
wait 2000ms
|
|
291
|
-
|
|
292
|
-
// 2. Observe stability
|
|
293
|
-
evaluate "window.location.href" // url2
|
|
294
|
-
check console for errors
|
|
295
|
-
|
|
296
|
-
// 3. Detect
|
|
297
|
-
if (url1 !== url2) {
|
|
298
|
-
→ ❌ URL changed — redirect loop
|
|
299
|
-
→ Is this route protected without valid auth?
|
|
300
|
-
→ Yes → auth.setup.ts is broken → fix auth first
|
|
301
|
-
→ No → App middleware bug → mark route ❌ skip, record as App Bug
|
|
302
|
-
}
|
|
303
|
-
if (console errors/warnings > 10) {
|
|
304
|
-
→ ❌ Excessive console errors — page refresh / JS crash loop
|
|
305
|
-
→ Mark route ❌ skip, record as App Bug
|
|
306
|
-
}
|
|
307
|
-
```
|
|
308
|
-
|
|
309
|
-
**Network monitoring**: After navigating, check network requests for failed API calls. Failed requests (status ≥ 400) on a route indicate an API/backend issue — record in `app-exploration.md` for reference.
|
|
310
|
-
|
|
311
|
-
**For guest routes** (no auth):
|
|
312
|
-
|
|
313
|
-
```
|
|
314
|
-
navigate <url>
|
|
315
|
-
```
|
|
316
|
-
|
|
317
|
-
**For protected routes** (auth required):
|
|
318
|
-
|
|
319
|
-
```
|
|
320
|
-
// Option A: use existing storageState (recommended)
|
|
321
|
-
// Option B: navigate to /login first, fill form, then navigate to target
|
|
322
|
-
// Option C: set auth cookies directly via document.cookie
|
|
323
|
-
```
|
|
107
|
+
**4.2. Explore each route**: navigate → check console errors → snapshot DOM → screenshot. For ≥5 routes, use `openspec-pw explore --parallel N` for genuine parallel browsers.
|
|
324
108
|
|
|
325
|
-
**
|
|
109
|
+
**App-level error decisions**:
|
|
326
110
|
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
111
|
+
| Signal | Action |
|
|
112
|
+
| ------ | ------ |
|
|
113
|
+
| HTTP 5xx / unreachable | **STOP** — backend error, fix app first |
|
|
114
|
+
| JS error in console | **STOP** — page has JS errors |
|
|
115
|
+
| HTTP 404 | Continue — mark `⚠️ route not found` |
|
|
116
|
+
| Auth required, no credentials | Continue — skip protected routes, explore login page |
|
|
117
|
+
| API 4xx/5xx | Continue — mark `⚠️ API error` |
|
|
330
118
|
|
|
331
|
-
|
|
119
|
+
**Redirect/Refresh loop detection**: Navigate → wait networkidle → capture URL → wait 2s → capture URL again. If URL changed → redirect loop (auth issue or middleware bug). If console errors > 10 → refresh loop. Skip route, record as App Bug.
|
|
332
120
|
|
|
333
|
-
|
|
334
|
-
- **Vue 2/3 / Angular / React 18 / Plain JS / jQuery**: `waitForSelector(targetElement)` is sufficient and faster — DOM updates are synchronous; Playwright's actionability checks auto-wait correctly
|
|
335
|
-
- Prefer specific element waits (`waitForSelector`) over generic load states
|
|
336
|
-
- Ready signal: heading, spinner disappears, or URL change
|
|
121
|
+
**Page wait stability**: React 19 / Next.js App Router → `page.waitForLoadState('networkidle')` (concurrent mode batches async). Vue/Angular/React 18/plain JS → `waitForSelector(targetElement)` (DOM updates synchronous). Prefer specific element waits over generic load states.
|
|
337
122
|
|
|
338
|
-
|
|
123
|
+
**Element extraction from snapshot**:
|
|
339
124
|
|
|
340
|
-
|
|
125
|
+
| Element type | Selector priority |
|
|
126
|
+
| ------------ | ----------------- |
|
|
127
|
+
| Buttons | `[data-testid]` > `getByRole` > `getByLabel` > `getByText` |
|
|
128
|
+
| Form fields | `[data-testid]` > `name` > `label` |
|
|
129
|
+
| Navigation | `text` > `href` |
|
|
130
|
+
| Headings, errors | For assertions |
|
|
131
|
+
| Special (canvas, iframe, CAPTCHA, OTP, Shadow DOM, file upload) | Detect & note strategy |
|
|
341
132
|
|
|
342
|
-
|
|
343
|
-
| -------------------- | ------------------------------------ | ---------------------------------------------------------- |
|
|
344
|
-
| **Buttons** | text, selector | `[data-testid]` > `getByRole` > `getByLabel` > `getByText` |
|
|
345
|
-
| **Form fields** | name, type, label, selector | `[data-testid]` > `name` > `label` |
|
|
346
|
-
| **Navigation links** | text, href, selector | `text` > `href` |
|
|
347
|
-
| **Headings** | text content, selector | for assertions |
|
|
348
|
-
| **Error messages** | text patterns, selector | for error path testing |
|
|
349
|
-
| **Dynamic content** | structure — row counts, card layouts | for data-driven tests |
|
|
350
|
-
| **Special elements** | type, selector, dimensions | for canvas/iframe/Shadow DOM test strategies |
|
|
133
|
+
**Special elements — quick reference**:
|
|
351
134
|
|
|
352
|
-
|
|
135
|
+
| Element | Snapshot signal | Strategy |
|
|
136
|
+
| ------- | --------------- | -------- |
|
|
137
|
+
| `<canvas>` | role="img" | `evaluate: getContext`, boundingBox > 0 |
|
|
138
|
+
| `<iframe>` | role="iframe" | frameLocator + src attr |
|
|
139
|
+
| CAPTCHA | `.g-recaptcha`, `[data-sitekey]` | auth.setup bypass / test.skip |
|
|
140
|
+
| OTP | 6-digit input fields, maxLength=1 | Dev bypass / E2E_OTP_CODE |
|
|
141
|
+
| Shadow DOM | role="generic" no children | `evaluate: el.shadowRoot` |
|
|
142
|
+
| Rich text | `[contenteditable]` | type + textContent |
|
|
143
|
+
| File upload | `<input type="file">` | setInputFiles |
|
|
353
144
|
|
|
354
|
-
|
|
145
|
+
Record findings in `app-exploration.md`. Output path:
|
|
146
|
+
- Change mode: `openspec/changes/<name>/specs/playwright/app-exploration.md`
|
|
147
|
+
- All mode: `<root>/app-exploration.md`
|
|
355
148
|
|
|
356
|
-
**
|
|
149
|
+
**Idempotency**: If `app-exploration.md` exists → read, verify routes, update only changed/new routes. **Route Snapshot Hash**: Navigate to sitemap.xml → hash content → if unchanged since last exploration, skip re-exploration entirely. Store hash in `app-knowledge.md` → `Exploration State`.
|
|
357
150
|
|
|
358
|
-
|
|
359
|
-
| ------- | --------------- | ---------------------------------------------- | ------------------- |
|
|
360
|
-
| `<canvas>` | `role="img"`, `tagName="CANVAS"` | `canvas.getContext('2d'/'webgl')`, `width`, `height` | High |
|
|
361
|
-
| `<iframe>` | `role="iframe"`, `src` attribute | `frameLocator` available | High |
|
|
362
|
-
| CAPTCHA | `.g-recaptcha`, `.h-captcha`, `[data-sitekey]`, canvas+slider | recaptcha score via API (if configured) | High |
|
|
363
|
-
| OTP / SMS | 6-digit input, countdown timer | Check if dev bypass exists | High |
|
|
364
|
-
| Shadow DOM | `role="generic"` with no children | Check `shadowRoot` via evaluate | Medium |
|
|
365
|
-
| Rich text editor | `[contenteditable]`, `role="textbox"` | `innerHTML`, `getContent()` | Medium |
|
|
366
|
-
| Video / Audio | `role="application"` or name contains "video"/"audio" | `evaluate` checks both `<video>` and `<audio>` tags | Medium |
|
|
367
|
-
| File upload | `<input type="file">` | `accept` attribute, `multiple` flag | Medium |
|
|
368
|
-
| Drag-and-drop | drag events in JS | Simulate DnD via coordinate clicks | Low |
|
|
369
|
-
| Date picker | specific `data-testid` or class patterns | Click triggers → evaluate value | Low (skip unless specs mention) |
|
|
370
|
-
| Infinite scroll | Dynamic row insertion | Count elements before/after scroll | Low (skip unless specs mention dynamic lists/pagination) |
|
|
371
|
-
| WebSocket / SSE | No DOM signal | Check page console for WS events | Low (check only if app uses real-time features) |
|
|
151
|
+
**4.3. Update shared knowledge**: Extract project-level findings to `tests/playwright/app-knowledge.md`. Auto-de-duplicate by key.
|
|
372
152
|
|
|
373
|
-
|
|
374
|
-
- Canvas: `getContext('webgl2'/'webgl'/'2d')`, `width`, `height`
|
|
375
|
-
- Iframe: `src` attribute → use `frameLocator` in tests
|
|
376
|
-
- CAPTCHA: `.g-recaptcha`, `.h-captcha`, `[data-sitekey]`, canvas+slider detection
|
|
377
|
-
- OTP: `input` elements with `maxLength === 1` or `type === 'tel'`
|
|
378
|
-
- Rich text: `[contenteditable]` → `innerHTML`, `textContent.length`
|
|
379
|
-
- Video/Audio: `querySelector('video'/'audio')` → `paused`, `duration`
|
|
380
|
-
- Shadow DOM: `role="generic"` with no children → check `shadowRoot`
|
|
153
|
+
### 5. Generate test plan (change mode only)
|
|
381
154
|
|
|
382
|
-
|
|
155
|
+
**All mode**: skip — show confirmation, proceed to Step 6.
|
|
383
156
|
|
|
384
|
-
|
|
157
|
+
**Prerequisite**: `app-exploration.md` must exist → STOP and run Step 4 if missing.
|
|
385
158
|
|
|
386
|
-
|
|
159
|
+
Create `openspec/changes/<name>/specs/playwright/test-plan.md`. Read inputs: specs, app-exploration.md, app-knowledge.md. Create test cases (functional requirement → test case, with `@role` and `@auth` tags). Reference verified selectors from exploration.
|
|
387
160
|
|
|
388
|
-
**
|
|
161
|
+
**State mutual exclusion**: before each test case, identify state boundaries and which elements disappear/appear. Assert mutual exclusion explicitly.
|
|
389
162
|
|
|
390
|
-
|
|
163
|
+
**Idempotency**: If test-plan.md exists → read and supplement missing cases, never regenerate.
|
|
391
164
|
|
|
392
|
-
- **
|
|
393
|
-
- **Auth**: none / required (storageState: `<path>`)
|
|
394
|
-
- **Ready signal**: how to know the page is loaded
|
|
395
|
-
- **Elements**: interactive elements with verified selectors (see 4.3 table)
|
|
396
|
-
- **Screenshot**: `__screenshots__/<slug>.png`
|
|
397
|
-
|
|
398
|
-
After exploration, add route-level notes (redirects, dynamic content → see 4.5).
|
|
399
|
-
|
|
400
|
-
#### 4.5. Exploration behavior notes
|
|
401
|
-
|
|
402
|
-
| Situation | Action |
|
|
403
|
-
| ------------------------------------------------- | ---------------------------------------------------------------- |
|
|
404
|
-
| SPA routing (URL changes but page doesn't reload) | Explore via navigation clicks from known routes, not direct URLs |
|
|
405
|
-
| Page loads but no interactive elements | Wait longer for SPA hydration |
|
|
406
|
-
| Dynamic content (user-specific) | Record structure — use `toContainText` or regex, not `toHaveText` |
|
|
407
|
-
|
|
408
|
-
**Idempotency**: If `app-exploration.md` already exists → read it, verify routes still match the live app, update only new routes or changed pages.
|
|
409
|
-
|
|
410
|
-
#### Route Snapshot Hash -- Skip Unchanged Routes
|
|
411
|
-
|
|
412
|
-
Before re-exploring, compute a lightweight hash of the app's current state:
|
|
413
|
-
|
|
414
|
-
1. **Quick hash** (for re-runs): Navigate to `${BASE_URL}/sitemap.xml` → hash the XML content
|
|
415
|
-
2. **If hash unchanged** since last exploration: Skip re-exploration entirely -- use cached `app-exploration.md`
|
|
416
|
-
3. **If hash changed**: Re-explore only changed routes (diff sitemap XML)
|
|
417
|
-
4. **Store hash** in `app-knowledge.md` → `Exploration State` section
|
|
418
|
-
|
|
419
|
-
```markdown
|
|
420
|
-
## Exploration State
|
|
421
|
-
Last explored: 2026-04-12T07:30:00Z
|
|
422
|
-
Sitemap hash: sha256:abc123...
|
|
423
|
-
Routes count: 20
|
|
424
|
-
```
|
|
425
|
-
|
|
426
|
-
This prevents redundant exploration on every `/opsx:e2e` run.
|
|
427
|
-
|
|
428
|
-
#### 4.6. Update app-knowledge.md
|
|
429
|
-
|
|
430
|
-
After writing `app-exploration.md`, extract **project-level shared knowledge** and append to `tests/playwright/app-knowledge.md`. **Auto-de-duplicate**: before adding any row, check if the same key already exists — if so, skip.
|
|
431
|
-
|
|
432
|
-
| Section | What to extract | De-duplication key |
|
|
433
|
-
| ------- | --------------- | ----------------- |
|
|
434
|
-
| Architecture | Monolith or separated? Backend port? Restart command? | **Section-level**: update existing rows instead of adding duplicates |
|
|
435
|
-
| Credential Format | Login endpoint, username format (email vs username) | **Field-level**: `Field` column (username, password, login endpoint) |
|
|
436
|
-
| Common Selector Patterns | New patterns discovered that apply across routes | **Element + Selector**: skip if same Element + Selector already exists |
|
|
437
|
-
| SPA Routing | SPA framework, routing behavior | **Section-level**: update existing instead of appending |
|
|
438
|
-
| Project Conventions | BASE_URL, auth method, multi-user roles | **Convention column**: skip if same convention already exists |
|
|
439
|
-
| Selector Fixes | Healed selectors (see Step 9 Phase 2-7) — route, old → new selector, reason, date | **Route + Old Selector**: skip if same key exists |
|
|
440
|
-
| Assertion Fixes | Healed assertions (see Step 9 Phase 2-7) — test, old → new assertion, reason, date | **Test + Old Assertion**: skip if same key exists |
|
|
441
|
-
|
|
442
|
-
Append only new/changed items — preserve existing content.
|
|
443
|
-
|
|
444
|
-
### 4.5. Vision Check (Optional, VLM-powered)
|
|
445
|
-
|
|
446
|
-
If Ollama with a vision model is available, analyze screenshots for layout anomalies before proceeding to test generation. This catches UI issues early — before tests fail.
|
|
447
|
-
|
|
448
|
-
**Prerequisite**: Run `openspec-pw doctor` to check Vision Check availability:
|
|
449
|
-
```
|
|
450
|
-
─── Vision Check ───
|
|
451
|
-
✓ ollama: http://localhost:11434 (qwen2.5-vl)
|
|
452
|
-
```
|
|
453
|
-
|
|
454
|
-
If Vision Check shows `⚠ disabled` or `⚠ not reachable` → skip this step entirely. Vision Check is optional and should not block exploration.
|
|
455
|
-
|
|
456
|
-
**Configuration**:
|
|
457
|
-
- `.env` file in `tests/playwright/` (highest priority): `OLLAMA_URL`, `OLLAMA_VISION_MODEL`, `OLLAMA_VISION_ENABLED`
|
|
458
|
-
- Environment variables: `OLLAMA_URL`, `OLLAMA_VISION_MODEL`
|
|
459
|
-
|
|
460
|
-
If no configuration is found, vision check is disabled.
|
|
461
|
-
|
|
462
|
-
**Run vision check** after exploration is complete:
|
|
463
|
-
|
|
464
|
-
```bash
|
|
465
|
-
openspec-pw vision-check --screenshots "__screenshots__/*.png"
|
|
466
|
-
```
|
|
467
|
-
|
|
468
|
-
**Flags**:
|
|
469
|
-
- `--screenshots <glob>` — Screenshot paths (required)
|
|
470
|
-
- `--parallel <n>` — Concurrent Ollama requests (default: 4)
|
|
471
|
-
- `--severity <levels>` — Filter by severity: `blocking,warning,minor`
|
|
472
|
-
- `--output <path>` — Write JSON results to file
|
|
473
|
-
- `--dry-run` — List screenshots without analyzing
|
|
474
|
-
- `--json` — Output JSON format
|
|
475
|
-
|
|
476
|
-
**Exit codes**:
|
|
477
|
-
- `0` = Check completed (with or without anomalies)
|
|
478
|
-
- `1` = Ollama not available → skip vision check, continue workflow
|
|
479
|
-
- `2` = Configuration disabled → skip vision check, continue workflow
|
|
480
|
-
|
|
481
|
-
**Anomaly types detected**:
|
|
482
|
-
|
|
483
|
-
| Type | Description | Example |
|
|
484
|
-
| --- | --- | --- |
|
|
485
|
-
| `obscured` | Interactive element covered by another element | Submit button hidden behind modal |
|
|
486
|
-
| `crowded` | Elements too close together, hard to distinguish | Form fields with <8px spacing |
|
|
487
|
-
| `overflowed` | Content clipped by container | Truncated text without ellipsis |
|
|
488
|
-
|
|
489
|
-
**Output handling**:
|
|
490
|
-
|
|
491
|
-
1. **JSON output** (with `--json`): Parse anomalies array for programmatic use
|
|
492
|
-
2. **Auto-append**: Without `--json`, anomalies are appended to `app-exploration.md` → **Visual Anomalies** table
|
|
493
|
-
3. **Decision table**:
|
|
494
|
-
|
|
495
|
-
| Anomaly Severity | Action |
|
|
496
|
-
| --- | --- |
|
|
497
|
-
| `blocking` | Flag route with `⚠️ Layout`, consider skipping test generation until fixed |
|
|
498
|
-
| `warning` | Log to Visual Anomalies table, continue test generation |
|
|
499
|
-
| `minor` | Log for reference, no action required |
|
|
500
|
-
|
|
501
|
-
**Graceful degradation**: If Ollama is unavailable or times out → log warning, skip vision check, continue to Step 5. Vision check should never block the E2E workflow.
|
|
502
|
-
|
|
503
|
-
#### 4.7. After exploration
|
|
504
|
-
|
|
505
|
-
Pass `app-exploration.md` to:
|
|
506
|
-
|
|
507
|
-
- **Step 5 (Planner)**: reference real routes, auth states, and elements in test-plan.md
|
|
508
|
-
- **Step 6 (Generator)**: use verified selectors instead of inferring
|
|
509
|
-
|
|
510
|
-
Read `tests/playwright/app-knowledge.md` as context for cross-change patterns.
|
|
511
|
-
|
|
512
|
-
### 5. Generate test plan
|
|
513
|
-
|
|
514
|
-
> **"all" mode: skip test-plan generation.** No OpenSpec specs → no test-plan to generate. Still show confirmation below, then proceed to Step 6.
|
|
515
|
-
|
|
516
|
-
**All mode — brief confirmation before Step 6:**
|
|
517
|
-
```
|
|
518
|
-
## All Mode: Page Object Discovery
|
|
519
|
-
Discovered <N> routes (<M> guest, <K> protected)
|
|
520
|
-
Special elements: <element summary>
|
|
521
|
-
Ready to generate Page Objects for: <page-name>Page.ts, <page-name>Page.ts, ...
|
|
522
|
-
Reply **yes** to proceed, or tell me to exclude routes or adjust strategies.
|
|
523
|
-
```
|
|
524
|
-
|
|
525
|
-
**Change mode — prerequisite**: If `openspec/changes/<name>/specs/playwright/app-exploration.md` does not exist → **STOP**. Run Step 4 (explore application) before generating tests. Without real DOM data from exploration, selectors are guesses and tests will be fragile. **All mode**: if `<root>/app-exploration.md` does not exist → same STOP.
|
|
526
|
-
|
|
527
|
-
**Change mode**: Create `openspec/changes/<name>/specs/playwright/test-plan.md`.
|
|
528
|
-
|
|
529
|
-
**Read inputs**: specs, app-exploration.md, app-knowledge.md
|
|
530
|
-
|
|
531
|
-
**Create test cases**: functional requirement → test case, with `@role` and `@auth` tags. Reference verified selectors from app-exploration.md.
|
|
532
|
-
|
|
533
|
-
> **State mutual exclusion**: before creating each test case — where are the state boundaries? During state transitions, which elements disappear or appear? Mutual exclusion must be asserted explicitly.
|
|
534
|
-
|
|
535
|
-
If a test case requires `page.route()` API mocking → append `⚠️ API Mock` flag to the test case line in the summary, with reason.
|
|
536
|
-
|
|
537
|
-
**Idempotency**: If test-plan.md exists → read and use, **but you MAY supplement missing test cases**. "Do not regenerate" means: do not discard existing cases, but you CAN add new ones discovered during Step 4 exploration that weren't in the original spec (e.g., empty states, error paths found during DOM exploration).
|
|
538
|
-
|
|
539
|
-
**⚠️ Human verification — STOP before generating code.**
|
|
540
|
-
|
|
541
|
-
After creating (or reading existing) test-plan.md, **stop and display the test plan summary** for user confirmation:
|
|
542
|
-
|
|
543
|
-
**Output format** — show the test plan in markdown directly in the conversation:
|
|
544
|
-
|
|
545
|
-
````markdown
|
|
546
|
-
## Test Plan Summary: `<change-name>`
|
|
547
|
-
|
|
548
|
-
**Auth**: required / not required | Roles: ...
|
|
549
|
-
|
|
550
|
-
### Test Cases
|
|
551
|
-
- ✅ `<test-name>` — `<route>`, happy path
|
|
552
|
-
- ✅ `<test-name>` — `<route>`, error path: `<error condition>`
|
|
553
|
-
|
|
554
|
-
### Special Elements
|
|
555
|
-
- ⚠️ **CAPTCHA** at `<route>` — strategy: `auth.setup bypass / skip / api-only`
|
|
556
|
-
- ⚠️ **Canvas/WebGL** at `<route>` — strategy: screenshot + dimensions
|
|
557
|
-
- ⚠️ **OTP** at `<route>` — strategy: test credentials / dev bypass
|
|
558
|
-
- ⚠️ **Iframe** at `<route>` — strategy: frameLocator + assert inner content
|
|
559
|
-
- ⚠️ **Video/Audio** at `<route>` — strategy: play() + assert !paused
|
|
560
|
-
- ⚠️ **File Upload** at `<route>` — strategy: setInputFiles + assert upload
|
|
561
|
-
- ⚠️ **Drag-and-Drop** at `<route>` — strategy: dragAndDrop or evaluate events
|
|
562
|
-
- ⚠️ **WebSocket/SSE** at `<route>` — strategy: waitForResponse + waitForFunction
|
|
563
|
-
|
|
564
|
-
### Not Covered
|
|
565
|
-
- `<element or scenario not testable>`
|
|
566
|
-
````
|
|
567
|
-
|
|
568
|
-
**Important**: Only list special elements that were actually detected in Step 4. Do not pre-populate with all possible types. If no special elements were found → omit the **Special Elements** section entirely.
|
|
569
|
-
|
|
570
|
-
Then ask: "Does this coverage match your intent? Reply **yes** to proceed, or tell me what to add/change."
|
|
571
|
-
|
|
572
|
-
If the user requests changes → update test-plan.md → re-display summary → re-confirm → proceed.
|
|
165
|
+
**⚠️ Human verification**: After creating/reading test-plan.md, **STOP** and display summary. Ask user to confirm before proceeding to Step 6.
|
|
573
166
|
|
|
574
167
|
### 6. Generate (Generator role)
|
|
575
168
|
|
|
576
|
-
**
|
|
577
|
-
|
|
578
|
-
**Prerequisite** (change mode only): If `openspec/changes/<name>/specs/playwright/app-exploration.md` does not exist → **STOP**. Run Step 4 first. For **all mode**, exploration is embedded dynamically in Step 6 — no pre-existing app-exploration.md is required.
|
|
579
|
-
|
|
580
|
-
**Page Object pattern** — read before writing any page file:
|
|
169
|
+
**All mode**: Build Page Objects for future tests. **Change mode**: Generate `tests/playwright/changes/<name>/<name>.spec.ts`.
|
|
581
170
|
|
|
171
|
+
**Page Object pattern** (read BasePage.ts first):
|
|
582
172
|
```typescript
|
|
583
|
-
// ✅ 正确:getters + async actions + this.click/fill
|
|
584
173
|
export class LoginPage extends BasePage {
|
|
585
174
|
get usernameInput() { return this.byLabel('用户名'); }
|
|
586
175
|
get submitBtn() { return this.byRole('button', { name: '登录' }); }
|
|
@@ -591,226 +180,43 @@ export class LoginPage extends BasePage {
|
|
|
591
180
|
await this.click(this.submitBtn);
|
|
592
181
|
}
|
|
593
182
|
}
|
|
594
|
-
|
|
595
|
-
// ❌ 错误:测试文件里写 inline locators
|
|
596
|
-
test('login', async ({ page }) => {
|
|
597
|
-
await page.getByLabel('用户名').fill('user'); // ← never do this!
|
|
598
|
-
});
|
|
599
183
|
```
|
|
600
184
|
|
|
601
|
-
**
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
| `pages/<Route>Page.ts` does not exist | Create from LoginPage pattern |
|
|
606
|
-
| File exists with some getters | Extend — add missing, preserve existing |
|
|
607
|
-
| File exists but uses inline locators | Rewrite with Page Object pattern, keep selector strings |
|
|
608
|
-
| Route removed from app | Remove corresponding Page Object file |
|
|
609
|
-
|
|
610
|
-
**File naming**: `pages/<Route>Page.ts` — use kebab-case route → PascalCase. `/login` → `LoginPage.ts`, `/user-profile` → `UserProfilePage.ts`.
|
|
611
|
-
|
|
612
|
-
For each discovered route:
|
|
613
|
-
|
|
614
|
-
1. Read existing `pages/<Route>Page.ts` (if any — incremental, not overwrite)
|
|
615
|
-
2. Navigate to route with correct auth state
|
|
616
|
-
3. browser_snapshot to extract interactive elements (see 4.3 table)
|
|
617
|
-
4. Write or update `pages/<Route>Page.ts` — extend with newly discovered elements
|
|
618
|
-
5. Also write `tests/playwright/app-all.spec.ts` — smoke test. **Minimum standard**: verify at least one heading or key interactive element is visible — not just "no crash". If the page loads but shows an empty shell or an error, the test must fail.
|
|
619
|
-
|
|
620
|
-
**Output priority**: Page Objects (`pages/*.ts`) are the primary asset. Smoke test is secondary. Existing Page Objects are never overwritten — only extended.
|
|
621
|
-
|
|
622
|
-
**Change mode** → `tests/playwright/changes/<name>/<name>.spec.ts` (functional):
|
|
185
|
+
**Page Object file handling**:
|
|
186
|
+
- File doesn't exist → Create
|
|
187
|
+
- File exists with getters → Extend (preserve existing, add missing)
|
|
188
|
+
- File has inline locators → Rewrite with Page Object pattern
|
|
623
189
|
|
|
624
|
-
-
|
|
625
|
-
- For each test case: verify selectors in real browser, then write Playwright code
|
|
626
|
-
|
|
627
|
-
**Per-assertion UI check** (before writing each assertion):
|
|
190
|
+
**Per-assertion check**:
|
|
628
191
|
```
|
|
629
192
|
Is this assertion about a visible UI result?
|
|
630
193
|
→ Yes → MUST use: expect(locator) with page selector
|
|
631
|
-
→ No →
|
|
632
|
-
→ Yes → page.request is acceptable (record reason)
|
|
633
|
-
→ No → This is a bug — rewrite with UI selector
|
|
634
|
-
```
|
|
635
|
-
**Never use page.request for assertions the user can see on screen.** If you wrote page.request.get() for a visible result → rewrite with expect(locator) from the browser snapshot.
|
|
636
|
-
|
|
637
|
-
**Selector verification (change mode)**:
|
|
638
|
-
|
|
639
|
-
1. Navigate to route with correct auth state
|
|
640
|
-
2. browser_snapshot to confirm page loaded
|
|
641
|
-
3. For each selector: verify from current snapshot (see 4.3 table for priority)
|
|
642
|
-
4. Write test code with verified selectors
|
|
643
|
-
5. If selector unverifiable → note for Healer (Step 9)
|
|
644
|
-
|
|
645
|
-
**Selector Caching — reuse Step 4 exploration results:**
|
|
646
|
-
|
|
647
|
-
After Step 4, verified selectors are stored in `app-exploration.md`. Before navigating to verify a selector in Step 6, check if the route already has verified selectors from Step 4.
|
|
648
|
-
|
|
649
|
-
```javascript
|
|
650
|
-
// Step 6: Before navigating to verify, check cached selectors
|
|
651
|
-
const cachedSelectors = appExploration.routes.find(r => r.path === routePath)?.elements
|
|
652
|
-
if (cachedSelectors && cachedSelectors.length > 0) {
|
|
653
|
-
// Use first-priority selector from cache instead of re-navigating
|
|
654
|
-
const selector = getFirstPrioritySelector(cachedSelectors)
|
|
655
|
-
// Only navigate to verify if selector is missing or marked "Fragile"
|
|
656
|
-
if (selector.stability !== 'Fragile') {
|
|
657
|
-
return selector // use cached, skip verification navigation
|
|
658
|
-
}
|
|
659
|
-
}
|
|
660
|
-
// Fallback: navigate + snapshot + verify
|
|
661
|
-
```
|
|
662
|
-
|
|
663
|
-
**Benefit**: For a 50-test case suite, this saves 30-50 redundant navigations (~2-5 minutes).
|
|
664
|
-
|
|
665
|
-
**File to read**: The selector caching uses data already stored in Step 4.4's `app-exploration.md` output — no new file needed.
|
|
666
|
-
|
|
667
|
-
**Test coverage — empty states**: For list/detail pages, explore the empty state. If the app shows a "no data" UI when the list is empty, generate a test to verify it. Empty states are often missing from specs but are real user paths.
|
|
668
|
-
|
|
669
|
-
**Test data fabrication**: never invent user accounts, product lists, or API responses. Follow §6 of the employee standards — explicitly ask the user for test data; use `TODO(user)` markers for approved placeholders; fail loudly rather than fabricate.
|
|
670
|
-
|
|
671
|
-
**Test coverage — special elements**: Check `app-exploration.md` → **Special Elements Detected** table. For each special element, generate tests using the following strategies:
|
|
672
|
-
- Canvas: screenshot + boundingBox → dimensions > 0, or 2D pixel verification
|
|
673
|
-
- WebGL: screenshot only (no pixel comparison — rendering varies)
|
|
674
|
-
- Iframe: `frameLocator` + assert inner content visible
|
|
675
|
-
- Rich text: `contenteditable` → type + `textContent` assertion
|
|
676
|
-
- Video/Audio: `play()` → assert `!paused`
|
|
677
|
-
- CAPTCHA/OTP/File upload/Drag-drop: See AI-Opaque Elements section in templates
|
|
678
|
-
|
|
679
|
-
**Test coverage — AI-opaque elements**: For CAPTCHA, OTP, slider CAPTCHA, file upload, and drag-drop — elements that Playwright cannot reliably automate:
|
|
680
|
-
|
|
681
|
-
1. Mark the element in `app-exploration.md` → **Special Elements Detected** table with type and automation strategy
|
|
682
|
-
2. Generate the test using the appropriate strategy:
|
|
683
|
-
- **CAPTCHA**: Bypass via `auth.setup.ts` storageState, or skip with `test.skip()`, or verify via API
|
|
684
|
-
- **OTP**: Use pre-verified test credentials (`E2E_OTP_CODE` env var), or development bypass flag
|
|
685
|
-
- **File upload**: Use `page.setInputFiles()` with fixture files
|
|
686
|
-
- **Drag-drop**: Use `page.dragAndDrop()` or `page.evaluate()` with custom event dispatching
|
|
687
|
-
3. If the element is truly non-automatable, write `test.skip()` with a comment explaining why, and mark with `/handoff` for manual testing
|
|
688
|
-
|
|
689
|
-
**Test coverage — performance**: Generate a Core Web Vitals test **only if** the OpenSpec spec or app-exploration.md specifies explicit performance targets (e.g., "LCP must be under 2s"). If no business target is defined, skip performance testing — hard-coded thresholds (lcp < 2500ms) produce false passes and add noise.
|
|
690
|
-
|
|
691
|
-
```typescript
|
|
692
|
-
// 🚫 Avoid for special elements:
|
|
693
|
-
await canvas.screenshot() // screenshot alone — no dimension/size assertion
|
|
694
|
-
await expect(canvas).toHaveScreenshot() // pixel-to-pixel comparison for WebGL
|
|
695
|
-
|
|
696
|
-
// ✅ Always:
|
|
697
|
-
const box = await canvas.boundingBox();
|
|
698
|
-
expect(box.width).toBeGreaterThan(0);
|
|
194
|
+
→ No → page.request acceptable (record reason)
|
|
699
195
|
```
|
|
700
196
|
|
|
701
|
-
**
|
|
702
|
-
|
|
703
|
-
- Follow `seed.spec.ts` structure
|
|
704
|
-
- Use `test.describe(...)` for grouping
|
|
705
|
-
- Each test: `test('描述性名称', async ({ page }) => { ... })`
|
|
706
|
-
- Prefer `data-testid` selectors (see 4.3 table)
|
|
197
|
+
**Selector caching**: Use already-verified selectors from `app-exploration.md`. Only navigate to verify if selector is missing or marked Fragile.
|
|
707
198
|
|
|
708
|
-
|
|
199
|
+
**Test data fabrication**: Never invent. Follow §6 of employee standards — ask user, use `TODO(user)` markers.
|
|
709
200
|
|
|
710
|
-
|
|
711
|
-
- `goto(path)` — navigation with configurable `waitUntil`
|
|
712
|
-
- `byRole(role, opts)`, `byLabel(label)`, `byPlaceholder(text)`, `byText(text)`, `byTestId(id)` — selector helpers in priority order (semantic → form → fallback)
|
|
713
|
-
- `click(locator)`, `fill(locator, value)`, `fillAndVerify(locator, value)` — safe interactions; use `fillAndVerify` when the next action depends on the value being committed
|
|
714
|
-
- `waitForToast(text?)`, `waitForLoad(spinnerSelector?)` — wait utilities
|
|
715
|
-
- `reload()` — page reload with hydration
|
|
201
|
+
**Test coverage — empty states**: For list/detail pages, explore and test empty state UI.
|
|
716
202
|
|
|
717
|
-
**
|
|
718
|
-
|
|
719
|
-
```typescript
|
|
720
|
-
// tests/playwright/pages/LoginPage.ts
|
|
721
|
-
import { BasePage } from './BasePage';
|
|
722
|
-
import type { Page } from '@playwright/test';
|
|
723
|
-
|
|
724
|
-
export class LoginPage extends BasePage {
|
|
725
|
-
get usernameInput() { return this.byLabel('用户名'); }
|
|
726
|
-
get passwordInput() { return this.byLabel('密码'); }
|
|
727
|
-
get submitBtn() { return this.byRole('button', { name: '登录' }); }
|
|
728
|
-
|
|
729
|
-
constructor(page: Page) { super(page); }
|
|
730
|
-
|
|
731
|
-
async login(user: string, pass: string) {
|
|
732
|
-
await this.goto('/login');
|
|
733
|
-
await this.fillAndVerify(this.usernameInput, user);
|
|
734
|
-
await this.fillAndVerify(this.passwordInput, pass);
|
|
735
|
-
await this.click(this.submitBtn);
|
|
736
|
-
}
|
|
737
|
-
}
|
|
738
|
-
```
|
|
739
|
-
|
|
740
|
-
```typescript
|
|
741
|
-
// tests/playwright/changes/<name>/<name>.spec.ts
|
|
742
|
-
import { LoginPage } from '../pages/LoginPage';
|
|
743
|
-
|
|
744
|
-
test('user can login', async ({ page }) => {
|
|
745
|
-
const loginPage = new LoginPage(page);
|
|
746
|
-
await loginPage.login('user@example.com', 'password123');
|
|
747
|
-
await loginPage.expectURL(/dashboard/);
|
|
748
|
-
});
|
|
749
|
-
```
|
|
750
|
-
|
|
751
|
-
**If a shared page object doesn't exist yet**: define it inline in the spec AND write it to `tests/playwright/pages/<PageName>.ts` so future tests can reuse it.
|
|
752
|
-
|
|
753
|
-
#### 6.2. Selector patterns
|
|
754
|
-
|
|
755
|
-
| Prefer (robust) | Avoid (fragile) |
|
|
756
|
-
| --- | --- |
|
|
757
|
-
| `getByRole`, `getByTestId`, `getByLabel` | CSS class (`'.notification-bell'`), CSS ID (`'#avatarBtn'`) |
|
|
758
|
-
| `waitForSelector(targetElement)` | hardcoded `200ms` / `500ms` delays |
|
|
759
|
-
|
|
760
|
-
See above for Page Object pattern, LoginPage example, and BasePage utilities.
|
|
761
|
-
|
|
762
|
-
If the file exists → diff against test-plan, add only missing test cases.
|
|
203
|
+
**Performance tests**: Only if spec or app-exploration.md defines explicit targets. No hard-coded thresholds.
|
|
763
204
|
|
|
764
205
|
### 7. Configure auth (if required)
|
|
765
206
|
|
|
766
|
-
- **API login**: Generate `auth.setup.ts` using `E2E_USERNAME`/`E2E_PASSWORD` + POST
|
|
767
|
-
- **UI login**: Generate
|
|
207
|
+
- **API login**: Generate `auth.setup.ts` using `E2E_USERNAME`/`E2E_PASSWORD` + POST
|
|
208
|
+
- **UI login**: Generate using browser form fill
|
|
768
209
|
- **Multi-user**: Separate `storageState` paths per role
|
|
769
210
|
|
|
770
|
-
|
|
211
|
+
Always use env vars, never hardcode. If auth.setup.ts exists → verify, update only if stale.
|
|
771
212
|
|
|
772
|
-
- If
|
|
773
|
-
- If the app uses **username** (alphanumeric + underscore) → use `test_user_001` (more universal)
|
|
774
|
-
- Check existing test files or login page to determine the format
|
|
775
|
-
- Always set credentials via environment variables — never hardcode
|
|
776
|
-
|
|
777
|
-
**Prompt user**:
|
|
778
|
-
|
|
779
|
-
```
|
|
780
|
-
Auth required. To set up:
|
|
781
|
-
1. Customize tests/playwright/credentials.yaml
|
|
782
|
-
2. Export: export E2E_USERNAME=xxx E2E_PASSWORD=yyy
|
|
783
|
-
3. Run auth: npx playwright test --project=setup
|
|
784
|
-
4. Then run tests: openspec-pw run <name> # skips to Step 9 directly (artifacts are reused)
|
|
785
|
-
```
|
|
786
|
-
|
|
787
|
-
**Idempotency**: If `auth.setup.ts` already exists → verify format, update only if stale.
|
|
788
|
-
|
|
789
|
-
**Post-auth re-exploration**: If Step 4 skipped protected routes due to missing auth, re-run exploration for those routes now that auth is configured. Navigate to each protected route with auth context → snapshot DOM → update `app-exploration.md`. Selectors verified now are better than guesses used during test generation.
|
|
213
|
+
**Post-auth re-exploration**: If Step 4 skipped protected routes, re-run exploration for those routes now.
|
|
790
214
|
|
|
791
215
|
### 8. Configure playwright.config.ts
|
|
792
216
|
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
If missing → generate a minimal `playwright.config.ts` with webServer, projects, and reporters.
|
|
796
|
-
|
|
797
|
-
**Auto-detect BASE_URL** (in priority order):
|
|
798
|
-
|
|
799
|
-
1. `process.env.BASE_URL` if already set
|
|
800
|
-
2. `tests/playwright/seed.spec.ts` → extract `BASE_URL` value
|
|
801
|
-
3. Read `vite.config.ts` (or `vite.config.js`) → extract `server.port` + infer protocol (`https` if `server.https`, else `http`)
|
|
802
|
-
4. Read `package.json` → `scripts.dev` or `scripts.start` → extract port from `--port` flag
|
|
803
|
-
5. **Python projects**: Read `pyproject.toml` or `settings.py` → extract `PORT` or `DEBUG` config
|
|
804
|
-
6. **Go projects**: Read `main.go` or `.env` → extract port from `http.ListenAndServe` or env vars
|
|
805
|
-
7. Fallback: `http://localhost:3000`
|
|
217
|
+
If missing → generate minimal config with webServer, projects, reporters. If exists → preserve all fields, add only missing webServer block.
|
|
806
218
|
|
|
807
|
-
**
|
|
808
|
-
|
|
809
|
-
1. `package.json` → scripts in order: `dev` → `start` → `serve` → `preview` → `npm run dev`
|
|
810
|
-
2. **Python projects**: Check for `uvicorn`, `flask run`, `python manage.py runserver`, `fastapi dev`
|
|
811
|
-
3. **Go projects**: Check for `air`, `reflex`, `fresh`, or custom dev scripts
|
|
812
|
-
|
|
813
|
-
If playwright.config.ts exists → READ first, preserve ALL existing fields, add only missing `webServer` block.
|
|
219
|
+
**BASE_URL auto-detect**: process.env.BASE_URL → seed.spec.ts → vite.config.ts → package.json scripts → fallback `http://localhost:3000`.
|
|
814
220
|
|
|
815
221
|
### 9. Execute tests
|
|
816
222
|
|
|
@@ -818,356 +224,90 @@ If playwright.config.ts exists → READ first, preserve ALL existing fields, add
|
|
|
818
224
|
openspec-pw run <name> [options]
|
|
819
225
|
```
|
|
820
226
|
|
|
821
|
-
|
|
822
|
-
|
|
823
|
-
| Option | Description |
|
|
824
|
-
| --- | --- |
|
|
825
|
-
| `-p, --project <name>` | Playwright project to run (e.g., user, admin) |
|
|
826
|
-
| `-t, --timeout <seconds>` | Test timeout in seconds (default: 300) |
|
|
827
|
-
| `--json` | Output results as JSON |
|
|
828
|
-
| `-g, --grep <pattern>` | Run only tests matching pattern |
|
|
829
|
-
| `--smoke` | Run only smoke tests (equivalent to `--grep @smoke`) |
|
|
830
|
-
| `-w, --workers <n>` | Number of parallel workers |
|
|
831
|
-
| `--app-bugs <n>` | Number of app bugs (skipped tests) |
|
|
832
|
-
| `--healed <n>` | Number of test bugs healed by Healer |
|
|
833
|
-
| `--raft <n>` | Number of RAFTs detected |
|
|
834
|
-
| `--escalated <n>` | Number of human escalations |
|
|
835
|
-
| `--headed` | Show browser during test run (default: headless) |
|
|
836
|
-
| `--update-snapshots` | Update screenshot baselines before running tests |
|
|
837
|
-
|
|
838
|
-
The CLI handles: server lifecycle, port mismatch, report generation.
|
|
839
|
-
|
|
840
|
-
If tests fail → use Playwright MCP tools to inspect UI, fix selectors, re-run.
|
|
841
|
-
|
|
842
|
-
**Browser visibility**: The Healer uses browser MCP tools (snapshot, screenshot, console messages) to inspect failures — no need for `--headed`. If you want to **watch the browser yourself** during debugging, add `--headed`: `openspec-pw run <name> --headed`.
|
|
843
|
-
|
|
844
|
-
**Healer MCP tools** (in order of use):
|
|
845
|
-
|
|
846
|
-
| Tool | Purpose |
|
|
847
|
-
| -------------------------- | ----------------------------------------------- |
|
|
848
|
-
| `browser_navigate` | Go to the failing test's page |
|
|
849
|
-
| `browser_snapshot` | Get page structure to find equivalent selectors |
|
|
850
|
-
| `browser_console_messages` | Diagnose JS errors that may cause failures |
|
|
851
|
-
| `browser_network_requests` | Diagnose backend/API failures (4xx/5xx) |
|
|
852
|
-
| `browser_take_screenshot` | Visually compare before/after fixes |
|
|
853
|
-
| `browser_run_code` | Execute custom fix logic (optional) |
|
|
227
|
+
Flags: `-p --project`, `-t --timeout`, `--json`, `-g --grep`, `--smoke`, `-w --workers`, `--headed`, `--update-snapshots`.
|
|
854
228
|
|
|
855
|
-
|
|
229
|
+
When tests fail → **Healer** (3 phases):
|
|
856
230
|
|
|
857
|
-
|
|
858
|
-
|
|
859
|
-
**Healer — Phase 1: Triage**
|
|
860
|
-
|
|
861
|
-
When a test fails, classify before attempting repair.
|
|
862
|
-
|
|
863
|
-
**Batch Failure Detection — run this FIRST when multiple tests fail:**
|
|
864
|
-
|
|
865
|
-
```
|
|
866
|
-
Collect ALL failing test names + their failure reasons.
|
|
867
|
-
Group by: same route + same action + same Playwright error type (e.g., both "element not found" with the **same** root cause confirmed by browser_console_messages showing no JS error and browser_network_requests showing no 4xx/5xx). If root cause differs (e.g., element A missing vs element B in different section), treat as **separate** groups.
|
|
868
|
-
If ≥2 tests fall into the same group:
|
|
869
|
-
→ Pause individual healing
|
|
870
|
-
→ Navigate to that route + perform the action manually
|
|
871
|
-
→ Check browser_console_messages + browser_network_requests
|
|
872
|
-
→ If console error or 4xx/5xx present:
|
|
873
|
-
→ This is an App Bug (backend/API change), NOT Test Bugs
|
|
874
|
-
→ Classify all tests in this group as App Bug
|
|
875
|
-
→ Skip all → record 1 App Bug in registry (not N bugs)
|
|
876
|
-
→ Skip the rest of individual Triage for this group
|
|
877
|
-
→ If timeout errors:
|
|
878
|
-
→ Retry one isolated: `npx playwright test --grep "<first-test-name>"` [--project <role>]
|
|
879
|
-
→ If isolated passes → **RAFT** (shared state coupling) → skip all → note RAFT in report
|
|
880
|
-
→ If isolated still fails → individual **Flaky** (each test independently flaky) → proceed with individual Triage for this group
|
|
881
|
-
→ If no console/network error and no timeout, but all still fail:
|
|
882
|
-
→ Likely a shared state issue → **RAFT** → skip all → note RAFT in report
|
|
883
|
-
Proceed with individual Triage only for tests NOT in a batch failure group.
|
|
884
|
-
```
|
|
885
|
-
|
|
886
|
-
**After Batch Detection, individual Triage:**
|
|
231
|
+
**Phase 1 — Triage**: Classify each failure before repairing.
|
|
887
232
|
|
|
888
233
|
| Failure Type | Signal | Classification | Action |
|
|
889
|
-
|
|
|
890
|
-
|
|
|
891
|
-
|
|
|
892
|
-
|
|
|
893
|
-
|
|
|
894
|
-
|
|
|
895
|
-
|
|
|
896
|
-
|
|
|
897
|
-
| **Timeout** | waitFor/evaluate timeout | **Flaky** | Retry isolated: `npx playwright test --grep "<test-name>"` (1×, not counted in heal attempts). If it passes isolated but fails in suite → **RAFT**. If it consistently times out → check framework: React 19 / Next.js App Router: add `page.waitForLoadState('networkidle')`. Vue/Angular/React 18 / Plain JS / jQuery: use `waitForSelector(targetElement)` instead of timeout tuning. |
|
|
898
|
-
| **Same test fails in suite, passes isolated** | — | **RAFT** | `test.skip()` in suite, note RAFT in report |
|
|
899
|
-
|
|
900
|
-
- **App Bug** → skip immediately (no healing needed) → record in App Bug Registry
|
|
901
|
-
- **Flaky** → retry once isolated
|
|
902
|
-
- **Test Bug / Ambiguous** → Phase 2
|
|
903
|
-
|
|
904
|
-
#### App Bug Registry
|
|
905
|
-
|
|
906
|
-
For every App Bug classified in Phase 1, record it in `openspec/reports/app-bug-registry.md` (create if missing):
|
|
907
|
-
|
|
908
|
-
```markdown
|
|
909
|
-
# App Bug Registry
|
|
910
|
-
|
|
911
|
-
<!-- Auto-generated. Do not edit manually. -->
|
|
912
|
-
|
|
913
|
-
## Active App Bugs
|
|
914
|
-
|
|
915
|
-
| # | Test | Route | Signal | First Detected | Status |
|
|
916
|
-
|---|------|-------|--------|---------------|--------|
|
|
917
|
-
| 1 | test-name | /route | net::ERR_CONNECTION_REFUSED | 2026-04-09 | open |
|
|
918
|
-
```
|
|
919
|
-
|
|
920
|
-
**Update rules**:
|
|
921
|
-
- **New App Bug**: Append new row, increment `#`
|
|
922
|
-
- **App Bug re-run and now passes**: Keep row, change status to `resolved` + add `Resolved` column with date
|
|
923
|
-
- **Keep all rows** (never delete) — the resolved count is the signal that bugs are being fixed
|
|
924
|
-
|
|
925
|
-
> **Why this matters**: `test.skip()` hides App Bugs from the pass/fail count. Without an explicit registry, "all tests passed" is a false positive when App Bugs exist. The registry makes the invisible visible.
|
|
926
|
-
|
|
927
|
-
> **Global attempt guard**: Each test has an independent heal counter (max 3 per test). If the same test enters Phase 2 more than once and reaches the cap each time → treat as "consecutive escalation without progress" → Phase 3 immediately.
|
|
928
|
-
|
|
929
|
-
> **Type ≠ Blame**: "Test Bug" means the assertion or selector is wrong — it does NOT mean "blame the test author." The test was generated from the spec. Root cause may be spec ambiguity, spec→test generation error, or app→spec deviation. Only a human can determine blame.
|
|
930
|
-
|
|
931
|
-
**Healer — Phase 2: Repair**
|
|
932
|
-
|
|
933
|
-
After Triage classifies failure as "Test Bug" or "Ambiguous":
|
|
934
|
-
|
|
935
|
-
**Phase 2-0 — Batch diagnosis** (no Playwright, fast):
|
|
936
|
-
For ALL Phase 2 failures in this run, do this simultaneously:
|
|
937
|
-
1. Read the failing test spec file(s) to understand what each test verifies
|
|
938
|
-
2. Read `app-knowledge.md` for any previous fixes or selector patterns
|
|
939
|
-
3. Read `app-knowledge.md` → **Common Selector Patterns** for project conventions
|
|
940
|
-
4. For each test, output:
|
|
941
|
-
```
|
|
942
|
-
TEST: <test-name>
|
|
943
|
-
ROUTE: <route>
|
|
944
|
-
ASSERTION: "<what the test expects>"
|
|
945
|
-
EXPECTED_BEHAVIOR: <from spec, one line>
|
|
946
|
-
KNOWN_FIX: <yes/no — from app-knowledge.md Selector Fixes table>
|
|
947
|
-
```
|
|
948
|
-
5. Classify each test:
|
|
949
|
-
- `ready-to-fix` — known fix exists, or selector pattern is clear
|
|
950
|
-
- `needs-assertion-fix` — assertion itself needs changing (typo, wrong expected value)
|
|
951
|
-
- `needs-phase3` — ambiguous, no clear fix path
|
|
952
|
-
- `needs-more-diagnosis` — need to see actual page to determine
|
|
953
|
-
|
|
954
|
-
**Skip remaining steps for `needs-phase3` tests** — go directly to Phase 3.
|
|
955
|
-
|
|
956
|
-
**For `needs-assertion-fix` tests** — go to Phase 2-1 (navigate + snapshot → assertion fix).
|
|
957
|
-
|
|
958
|
-
**For `ready-to-fix` tests** — go to Phase 2-5 (selector repair).
|
|
959
|
-
|
|
960
|
-
**For `needs-more-diagnosis` tests** — go to Phase 2-1 (navigate + snapshot) for each individually.
|
|
961
|
-
|
|
962
|
-
**Phase 2-1** (for `needs-assertion-fix` or `needs-more-diagnosis`): Navigate to the failing page → `browser_snapshot` → **EXPLICIT COMPARISON**:
|
|
963
|
-
```
|
|
964
|
-
ASSERTION: "<what the test expects>"
|
|
965
|
-
ACTUAL: "<what the snapshot shows>"
|
|
966
|
-
MATCH: <yes/no>
|
|
967
|
-
```
|
|
968
|
-
- **If MATCH=yes** → apply the assertion fix → go to **Phase 2-6**
|
|
969
|
-
- **If MATCH=no**:
|
|
970
|
-
- Safe to fix without Phase 3 → apply the fix → go to **Phase 2-6**
|
|
971
|
-
- "Never fix without Phase 3" conditions met → **Phase 3** immediately
|
|
972
|
-
|
|
973
|
-
**Assertion modification guard — never skip Phase 3 unless ALL conditions are met:**
|
|
974
|
-
- The test has **never passed with the current assertion** (newly generated test, or this is the first time this assertion fails)
|
|
975
|
-
- The ACTUAL value is **verifiably** from a different spec section (e.g., this test was in the wrong describe block)
|
|
976
|
-
- You can point to the **specific line in the spec** that defines the ACTUAL behavior
|
|
234
|
+
| ------------ | ------ | -------------- | ------ |
|
|
235
|
+
| Network/Backend | `net::ERR`, 4xx/5xx | **App Bug** | `test.skip()` + App Bug Registry |
|
|
236
|
+
| JS Runtime Error | Console error | **App Bug** | `test.skip()` + App Bug Registry |
|
|
237
|
+
| Auth Expired | Redirected to /login | **Flaky** | Re-run auth.setup |
|
|
238
|
+
| Selector Not Found | Element not found | **Test Bug** | → Phase 2 |
|
|
239
|
+
| Assertion Mismatch | Wrong content | **Ambiguous** | → Phase 2 |
|
|
240
|
+
| Timeout | waitFor timeout | **Flaky** | Retry isolated |
|
|
241
|
+
| Same test: fails in suite, passes isolated | — | **RAFT** | `test.skip()` in suite, note in report |
|
|
977
242
|
|
|
978
|
-
**If
|
|
243
|
+
**Batch Detection**: When ≥2 tests fail with same route + error type, check console/network first. If backend error → all App Bug (1 entry). If timeout → check RAFT. If all same root cause → bulk classify.
|
|
979
244
|
|
|
980
|
-
**
|
|
981
|
-
- Typo in assertion (e.g., "Subm it" vs "Submit" in the expected text)
|
|
982
|
-
- Selector was correct but the element was moved to a different location (same text, different selector)
|
|
983
|
-
- Explicit spec drift confirmed by reading the spec (e.g., spec says "button says Submit" but test says "button says Submit Form")
|
|
245
|
+
**App Bug Registry**: `openspec/reports/app-bug-registry.md`. For each App Bug, append row (#, Test, Route, Signal, Date, Status). Never delete rows — resolved bugs change status to `resolved`.
|
|
984
246
|
|
|
985
|
-
**
|
|
986
|
-
- App behavior changed after an action (e.g., "after clicking submit, balance should decrease" → ACTUAL shows no change → **Phase 3**, could be optimistic update bug, backend failure, or spec mismatch)
|
|
987
|
-
- Data values differ (e.g., expected "¥1000" but got "¥999" → **Phase 3**, could be rounding, discount, or calculation bug)
|
|
988
|
-
- Missing elements after interaction (e.g., "after creating order, success message should appear" → no message → **Phase 3**)
|
|
247
|
+
**Phase 2 — Repair** (for Test Bug / Ambiguous):
|
|
989
248
|
|
|
990
|
-
|
|
249
|
+
2-0. **Batch diagnosis**: Read failing test specs + app-knowledge.md fixes. Output per test: TEST, ROUTE, ASSERTION, EXPECTED_BEHAVIOR, KNOWN_FIX. Classify: `ready-to-fix` / `needs-assertion-fix` / `needs-phase3` / `needs-more-diagnosis`.
|
|
991
250
|
|
|
992
|
-
|
|
251
|
+
2-1. **Navigate + snapshot**: ASSERTION vs ACTUAL comparison. Safe to fix without Phase 3: typo, moved element, confirmed spec drift. **Never fix without Phase 3**: behavior changed after action, data values differ, missing elements after interaction.
|
|
993
252
|
|
|
994
|
-
|
|
253
|
+
2-5. **Selector repair**: Pick highest-stability candidate: `getByRole` > `getByText/getByLabel` > `locator('#id')` > `locator('.class')` > `locator('nth-child')`.
|
|
995
254
|
|
|
996
|
-
|
|
255
|
+
2-6. **Fix + verify**: Apply fix → `npx playwright test --grep "<test-name>"` → if passes, log to app-knowledge.md (Selector Fixes / Assertion Fixes). Max 3 heal attempts per test.
|
|
997
256
|
|
|
998
|
-
|
|
999
|
-
Target: <element from failing assertion, e.g. "button with text 'Submit'">
|
|
1000
|
-
Candidates (stable → fragile):
|
|
1001
|
-
- getByRole(button, { name: 'Submit' }) ← Stable (semantic)
|
|
1002
|
-
- getByText('Submit') ← Fair (unique text)
|
|
1003
|
-
- getByLabel('Email') ← Fair (form fields)
|
|
1004
|
-
- locator('#submit') ← Fair (id attribute)
|
|
1005
|
-
- locator('.btn-primary') ← Fragile (style class) — upgrade if listed in Common Selector Patterns
|
|
1006
|
-
- locator('button:nth-child(3)') ← Fragile (DOM order)
|
|
1007
|
-
```
|
|
1008
|
-
|
|
1009
|
-
Stability: `getByRole` > `getByText`/`getByLabel` > `locator('#id')` > `locator('.class')` > `locator('nth-child')`. Upgrade stability if `app-knowledge.md` → **Common Selector Patterns** explicitly lists the selector as preferred for this project.
|
|
1010
|
-
|
|
1011
|
-
**Phase 2-5b. Select top candidate** — pick the highest-stability candidate that matches the target. Output: `SELECTED: <selector> — reason: <why this one>`. Then → go to **Phase 2-6**.
|
|
1012
|
-
|
|
1013
|
-
**Phase 2-6** — **Fix and verify incrementally (per test)**:
|
|
1014
|
-
1. For each test needing a fix, apply the fix to the `.spec.ts` file
|
|
1015
|
-
2. Run: `npx playwright test --grep "<test-name>"` to verify that specific test
|
|
1016
|
-
3. If passed → Phase 2-7 (log to app-knowledge.md), then move to next test
|
|
1017
|
-
4. If failed → analyze the new failure type:
|
|
1018
|
-
- **Assertion-related** (wrong value/content) → return to **Phase 2-1** for that test
|
|
1019
|
-
- **Selector-related** (element not found, wrong element) → return to **Phase 2-5** for that test (re-diagnose selector, not Phase 2-1)
|
|
1020
|
-
- **Timeout-related** → treat as Flaky → retry isolated once: `npx playwright test --grep "<test-name>"` [--project <role>]
|
|
1021
|
-
- If isolated passes → **Phase 2-7** (healed), then move to next test
|
|
1022
|
-
- If isolated still fails → apply framework fix (React 19 / Next.js: `page.waitForLoadState('networkidle')`; Vue/Angular/React 18 / Plain JS: `waitForSelector(targetElement)`) → retry → if passes → **Phase 2-7** (healed) → move to next test; if still fails → `test.skip()` in `.spec.ts` + note in report → move to next test (not counted in heal attempts)
|
|
1023
|
-
The heal counter does NOT reset for any path.
|
|
1024
|
-
5. Repeat until all Phase 2 tests are healed.
|
|
1025
|
-
|
|
1026
|
-
> **Why --grep instead of --only-changed**: Healer fixing is incremental — you fix one test, verify it, adjust if needed, then move to the next. `--only-changed` is **file-level**: it runs ALL tests in `.spec.ts` files that have uncommitted source changes, not individual tests. This wastes time in an iterative fix cycle (re-running already-passed tests in the same file). Use `--grep` for targeted verification. Reserve `--only-changed` for the pre-commit guard.
|
|
1027
|
-
|
|
1028
|
-
> **Pre-commit guard — `--only-changed`** (after all Phase 2 tests are healed): `npx playwright test --only-changed` runs all tests in changed `.spec.ts` files (comparing against HEAD by default, or `--only-changed=main` against a branch). Playwright also analyzes source file dependencies — if `src/components/Button.ts` changed, any `.spec.ts` that imports it will run. CI still runs the full suite as complete regression.
|
|
1029
|
-
|
|
1030
|
-
**Phase 2-7** — If healed → append to `app-knowledge.md` (auto-de-duplicate):
|
|
1031
|
-
|
|
1032
|
-
**Selector Fixes** — Before appending, read the existing **Selector Fixes** table. **Skip if** the same Route + Old Selector key already has a row (same route + same old selector = duplicate). If the Route + Old Selector exists but with a different New Selector → this is a new symptom of the same root cause → add a new row (different date, updated new selector).
|
|
1033
|
-
|
|
1034
|
-
**Assertion Fixes** — Before appending, read the existing **Assertion Fixes** table. **Skip if** the same Test + Old Assertion key already has a row. If the same test had a previous fix with a different old → new → add a new row (the assertion has drifted further).
|
|
1035
|
-
|
|
1036
|
-
If a new row is added, update the `Last updated` timestamp in the file header.
|
|
1037
|
-
|
|
1038
|
-
This log feeds the **Auto-Heal Log** in the Phase 10 report.
|
|
1039
|
-
|
|
1040
|
-
**Element Missing handling (when browser_snapshot shows element not found):**
|
|
1041
|
-
|
|
1042
|
-
| Situation | Check | Action |
|
|
1043
|
-
| --- | --- | --- |
|
|
1044
|
-
| JS error in console after action | `browser_console_messages` | **App Bug** → Phase 1 → App Bug classification |
|
|
1045
|
-
| Auth redirected mid-action | URL changed to `/login` | **Flaky** → re-run with fresh auth |
|
|
1046
|
-
| SPA route didn't update | URL is correct but element missing | Wait for SPA hydration → `page.waitForLoadState('networkidle')` or `waitForSelector(target)` |
|
|
1047
|
-
| Element genuinely missing | None of the above | **Test Bug** → find alternative selector or **Phase 3** if no equivalent exists |
|
|
1048
|
-
|
|
1049
|
-
**Healer — Phase 3: Escalate**
|
|
1050
|
-
|
|
1051
|
-
When Phase 2 tried ≥3 heals without success, OR ASSERTION vs ACTUAL comparison is ambiguous:
|
|
1052
|
-
|
|
1053
|
-
**STOP** and output:
|
|
257
|
+
2-7. **Log**: Append healed selector/assertion to `app-knowledge.md`. Auto-de-duplicate by Route + Old Selector (selectors) or Test + Old Assertion (assertions).
|
|
1054
258
|
|
|
259
|
+
**Phase 3 — Escalate** (after ≥3 heals or ambiguous comparison): STOP and output:
|
|
1055
260
|
```
|
|
1056
261
|
E2E Test Failed — Human Decision Required
|
|
1057
|
-
|
|
1058
|
-
|
|
1059
|
-
|
|
1060
|
-
Assertion: "<what test expects>"
|
|
1061
|
-
Actual: "<what app shows>"
|
|
1062
|
-
|
|
1063
|
-
This failure could be:
|
|
1064
|
-
1. App does not match the spec → **app bug**
|
|
1065
|
-
2. Test was generated from ambiguous/incorrect spec → **spec issue**
|
|
1066
|
-
3. Spec itself is outdated (app was updated) → **spec drift**
|
|
1067
|
-
|
|
1068
|
-
Please decide:
|
|
1069
|
-
(a) Fix the app to match the spec
|
|
1070
|
-
(b) Update the spec to match the app
|
|
1071
|
-
(c) Update the test assertion
|
|
1072
|
-
(d) Skip this test with test.skip() until resolved
|
|
262
|
+
Test: <name> | Failure: <type>
|
|
263
|
+
Assertion: "<expected>" | Actual: "<actual>"
|
|
264
|
+
Options: (a) Fix app, (b) Update spec, (c) Update assertion, (d) Skip
|
|
1073
265
|
```
|
|
1074
266
|
|
|
1075
|
-
Wait for user input
|
|
267
|
+
Wait for user input. Track escalation attempts — if 3 consecutive Phase 3 with no progress, STOP and flag.
|
|
1076
268
|
|
|
1077
|
-
**
|
|
269
|
+
**Post-heal**: After all Phase 2 tests healed, run `npx playwright test --only-changed` as pre-commit guard.
|
|
1078
270
|
|
|
1079
|
-
|
|
1080
|
-
|--------|-----------|-------------------|
|
|
1081
|
-
| **(a)** Fix the app to match the spec | Fix the app code | Re-run: `openspec-pw run <change-name>` to verify fix |
|
|
1082
|
-
| **(b)** Update the spec to match the app | Edit the spec file | Then update the test assertion (→ option c), or regenerate the affected part of the test |
|
|
1083
|
-
| **(c)** Update the test assertion | Fix the assertion in `tests/playwright/changes/<name>/<name>.spec.ts` | Re-run: `openspec-pw run <change-name>` to verify |
|
|
1084
|
-
| **(d)** Skip with `test.skip()` | Add `test.skip()` to the test | Note in `app-knowledge.md` → `Assertion Fixes` with reason "human escalation — skipped pending resolution" |
|
|
271
|
+
### 10. Report results
|
|
1085
272
|
|
|
1086
|
-
|
|
273
|
+
Compile from `playwright-e2e-<name>-<timestamp>.md` and Phase 1–3 output:
|
|
274
|
+
- Summary table (App Bugs / Test Bugs healed / Flaky-RAFT / Escalations)
|
|
275
|
+
- App Bug Summary (with accumulation warning if ≥3 active)
|
|
276
|
+
- Failure Classification table
|
|
277
|
+
- Auto-heal log
|
|
278
|
+
- RAFT Summary
|
|
279
|
+
- Human Escalations
|
|
280
|
+
- Recommendations
|
|
1087
281
|
|
|
1088
|
-
|
|
1089
|
-
|
|
1090
|
-
|
|
1091
|
-
|
|
1092
|
-
`/opsx:e2e <change-name>` re-runs the full 10-step workflow — unnecessary after Phase 3. The test file and auth context are already correct. Use `openspec-pw run` to verify fixes directly.
|
|
1093
|
-
|
|
1094
|
-
### 10. False Pass Detection + RAFT Detection + App Bug Accumulation + Report results
|
|
1095
|
-
|
|
1096
|
-
Run after test suite completes (even if all pass).
|
|
1097
|
-
|
|
1098
|
-
**False Pass patterns** (test passed but shouldn't have):
|
|
1099
|
-
|
|
1100
|
-
- **Conditional visibility**: `if (locator.isVisible().catch(() => false))` — if test passes, locator may not exist
|
|
1101
|
-
- **Too fast**: < 200ms for a complex flow is suspicious
|
|
1102
|
-
- **No fresh auth context**: Protected routes without `browser.newContext()`
|
|
1103
|
-
|
|
1104
|
-
**RAFT detection** (Resource-Affected Flaky Test):
|
|
1105
|
-
|
|
1106
|
-
- If you already ran the suite and a test failed: re-run that test in isolation with `npx playwright test --grep "<test-name>"` [--project <role>] → if it passes in isolation but fails in suite → **RAFT**
|
|
1107
|
-
- This is **NOT** a test bug or app bug. Mark as RAFT, add `test.skip()` in suite, note in report
|
|
1108
|
-
- RAFTs are infrastructure coupling issues (CPU/memory/I/O contention), not fixable by changing test or app
|
|
282
|
+
**Conditional "All Pass"**:
|
|
283
|
+
- ✅ **"All tests passed"** — 0 active App Bugs, 0 skipped
|
|
284
|
+
- ⚠️ **"All tests passed (N skipped)"** — skipped exist, no active App Bugs
|
|
285
|
+
- ⚠️ **"All tests passed (N skipped, M App Bugs unresolved)"** — active App Bugs exist
|
|
1109
286
|
|
|
1110
|
-
**App Bug
|
|
1111
|
-
|
|
1112
|
-
1. Read `openspec/reports/app-bug-registry.md`
|
|
1113
|
-
2. Count `open` status rows
|
|
1114
|
-
3. If ≥ 3 active App Bugs → add "⚠️ App Bug accumulation: N bugs unresolved" to report summary. **Do NOT suppress or hide this warning.** The test suite may report "all passed" (skipped ≠ failed), but N broken features is not a passing system.
|
|
1115
|
-
4. Check for **App Bugs that became "resolved"** (passing on re-run) → this is the signal that bugs were fixed. Update `app-bug-registry.md` accordingly, and remove `test.skip()` from those tests so they re-enter the regression suite.
|
|
1116
|
-
|
|
1117
|
-
**Report results** — compile from these sources (the auto-generated `playwright-e2e-<name>-<timestamp>.md` provides the Summary table; populate the sections below manually from Phase 1–3 output):
|
|
1118
|
-
|
|
1119
|
-
- **Summary table** with failure type breakdown (App Bugs, Test Bugs/healed, Flaky-RAFT, Human Escalations) ← from `openspec/reports/playwright-e2e-<name>-<timestamp>.md`
|
|
1120
|
-
- **App Bug Summary**: Table of all active App Bugs — test name, route, signal, first detected. If ≥ 3 active → display "⚠️ App Bug accumulation warning" prominently. ← from `openspec/reports/app-bug-registry.md`
|
|
1121
|
-
- **Conditional "All Pass" conclusion**:
|
|
1122
|
-
- ✅ **"All tests passed"** — only if 0 active App Bugs and 0 skipped tests
|
|
1123
|
-
- ⚠️ **"All tests passed (N skipped)"** — if skipped tests exist but no active App Bugs
|
|
1124
|
-
- ⚠️ **"All tests passed (N skipped, M App Bugs unresolved)"** — if active App Bugs exist
|
|
1125
|
-
- **Failure Classification table**: (test, type, action, healed?) ← compile from Phase 1 Triage + Phase 2-0 classification output
|
|
1126
|
-
- **Auto-heal log**: (assertion vs actual comparison, fix applied, result) ← compile from Phase 2-1/2-5 console output + `app-knowledge.md` Selector Fixes table
|
|
1127
|
-
- **RAFT Summary** (if any): ← compile from RAFT detection notes during Phase 1
|
|
1128
|
-
- **Human Escalations** (if any, with user decision): ← from Phase 3 decision output
|
|
1129
|
-
- **Recommendations** with `file:line` references
|
|
1130
|
-
|
|
1131
|
-
**Update tasks.md**:
|
|
1132
|
-
- If 0 active App Bugs → find E2E-related items, append `✅ Verified via Playwright E2E (<timestamp>)`.
|
|
1133
|
-
- If active App Bugs exist → **do not mark as verified**. Append instead: `⚠️ App Bug blocked: <bug summary> (<timestamp>)`.
|
|
287
|
+
**Update tasks.md**: If 0 active App Bugs → append `✅ Verified via Playwright E2E (<timestamp>)`. If App Bugs exist → append `⚠️ App Bug blocked: <summary> (<timestamp>)` (do not mark as verified).
|
|
1134
288
|
|
|
1135
289
|
## Graceful Degradation
|
|
1136
290
|
|
|
1137
|
-
| Scenario | Classification | Action |
|
|
1138
|
-
|
|
|
1139
|
-
| No specs / app-exploration
|
|
1140
|
-
| JS errors or HTTP 5xx during exploration | Blocker | **STOP**
|
|
1141
|
-
| Redirect
|
|
1142
|
-
| File already exists
|
|
1143
|
-
| Test fails (network/backend) | App Bug | `test.skip()` +
|
|
1144
|
-
| Test fails (selector/assertion) | Test Bug
|
|
1145
|
-
| RAFT detected
|
|
1146
|
-
| Phase 3 escalation | Human needed | **STOP**
|
|
1147
|
-
|
|
|
1148
|
-
| App Bug skip accumulation (≥3 active App Bugs) | Warning | Add "⚠️ App Bug accumulation: N bugs unresolved" to report. Do not suppress. | Continues |
|
|
1149
|
-
|
|
1150
|
-
**"STOP"** = workflow halts and waits for user input (true stop).
|
|
1151
|
-
**"Continues"** = workflow proceeds to next step (test-level skip, no user wait).
|
|
291
|
+
| Scenario | Classification | Action |
|
|
292
|
+
| -------- | -------------- | ------ |
|
|
293
|
+
| No specs / app-exploration missing (change mode) | Blocker | **STOP** |
|
|
294
|
+
| JS errors or HTTP 5xx during exploration | Blocker | **STOP** |
|
|
295
|
+
| Redirect/refresh loop during exploration | App Bug | Skip route → record in registry |
|
|
296
|
+
| File already exists | Idempotency | Read and use — never regenerate |
|
|
297
|
+
| Test fails (network/backend) | App Bug | `test.skip()` + registry |
|
|
298
|
+
| Test fails (selector/assertion) | Test Bug | Healer Phase 1→2 (≤3 attempts) |
|
|
299
|
+
| RAFT detected | Flaky | `test.skip()` in suite |
|
|
300
|
+
| Phase 3 escalation | Human needed | **STOP** — wait for user |
|
|
301
|
+
| ≥3 active App Bugs | Warning | Add accumulation warning to report |
|
|
1152
302
|
|
|
1153
303
|
## Guardrails
|
|
1154
304
|
|
|
1155
|
-
**Decision table:**
|
|
1156
|
-
|
|
1157
305
|
| Rule | Why |
|
|
1158
|
-
|
|
|
306
|
+
| ---- | --- |
|
|
1159
307
|
| Read specs as source of truth | Generated tests must match requirements |
|
|
1160
308
|
| Step 4 before Step 6 | Real DOM data → accurate selectors |
|
|
1161
309
|
| Never contradict specs | E2E validates implementation, not design |
|
|
1162
|
-
| Cap heal at 3 attempts | Prevents infinite loops |
|
|
310
|
+
| Cap heal at 3 attempts per test | Prevents infinite loops |
|
|
1163
311
|
| Write runnable code, not TODOs | Placeholders fail CI |
|
|
1164
312
|
|
|
1165
|
-
**
|
|
1166
|
-
`tests/playwright/`, `openspec/changes/<name>/specs/playwright/`, `openspec/reports/`, `playwright.config.ts`, `auth.setup.ts`
|
|
1167
|
-
|
|
1168
|
-
> `tests/playwright/` — spec files, Page Objects, auth, credentials, app-knowledge.md
|
|
1169
|
-
> `openspec/changes/<name>/specs/playwright/` — app-exploration.md, test-plan.md (change mode)
|
|
1170
|
-
> `<root>/` — app-exploration.md (change mode: INPUT for `openspec-pw explore`; all mode: detailed output)
|
|
1171
|
-
> `openspec/reports/` — test reports, app-bug-registry.md
|
|
1172
|
-
|
|
1173
|
-
**Never write to:** any other directory
|
|
313
|
+
**Write scope**: `tests/playwright/` (specs, Page Objects, auth, credentials, app-knowledge.md), `openspec/changes/<name>/specs/playwright/` (exploration, test plan), `<root>/app-exploration.md` (all mode), `openspec/reports/` (reports, bug registry), `playwright.config.ts`, `auth.setup.ts`. **Never write to any other directory.**
|