openspec-playwright 0.3.45 → 0.3.46

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/package.json +1 -1
  2. package/templates/e2e-command.md +159 -1019
@@ -11,576 +11,165 @@
11
11
  - **Page Objects** (all mode): `tests/playwright/pages/<Route>Page.ts`
12
12
  - **Auth setup**: `tests/playwright/auth.setup.ts` (if auth required)
13
13
  - **Report**: `openspec/reports/playwright-e2e-<name>-<timestamp>.md`
14
- - **App Bug Registry**: `openspec/reports/app-bug-registry.md` (cumulative, per-project)
14
+ - **App Bug Registry**: `openspec/reports/app-bug-registry.md` (cumulative)
15
15
  - **Test plan**: `openspec/changes/<name>/specs/playwright/test-plan.md` (change mode only)
16
16
 
17
17
  ## Architecture
18
18
 
19
- Two modes, same pipeline:
19
+ | Mode | Command | Route source | Output |
20
+ | ------ | ------------------- | ------------------------ | ------------------------------- |
21
+ | Change | `/opsx:e2e <name>` | OpenSpec specs | `changes/<name>/<name>.spec.ts` |
22
+ | All | `/opsx:e2e all` | sitemap + homepage crawl | `pages/*.ts` (Page Objects) |
20
23
 
21
- | Mode | Command | Route source | Output |
22
- | ------ | ------------------ | ------------------------ | ------------------------------- |
23
- | Change | `/opsx:e2e <name>` | OpenSpec specs | `changes/<name>/<name>.spec.ts` |
24
- | All | `/opsx:e2e all` | sitemap + homepage crawl | `pages/*.ts` (Page Objects) |
24
+ > **Full regression is opt-in only.** `openspec-pw run <name>` → one spec file. Do NOT run `npx playwright test` (no file) or `--only-changed` unless user explicitly requests.
25
+ > **Roles**: Planner (Steps 4–5) test-plan.md; Generator (Step 6) → `.spec.ts` + Page Objects; Healer (Step 9) → repairs failures via MCP.
25
26
 
26
- Both modes update `app-knowledge.md` and `app-exploration.md`. Spec files are independent per change — `openspec-pw run <name>` runs only `changes/<name>/<name>.spec.ts`.
27
+ Browser exploration is tool-agnostic: gstack (`/browse`), Playwright MCP, or `openspec-pw explore --parallel N`.
27
28
 
28
- > **⚠️ Full regression is opt-in only.** Default: `openspec-pw run <name>` → one spec file. Do NOT run `npx playwright test` (no file), `--only-changed`, or any command that executes multiple `.spec.ts` files unless the user explicitly requests it. This includes running the same command twice across different changes to simulate regression.
29
+ ## Testing Principles
29
30
 
30
- > **Role mapping**: Planner (Step 4–5) test-plan.md; Generator (Step 6) → `.spec.ts` + Page Objects; Healer (Step 9) repairs failures via MCP.
31
+ **UI first.** Every assertion about visible UI state must use `page.getByRole/ByLabel/ByText + expect()`. `page.request` is acceptable only for precondition setup or HTTP-level mocking via `page.route()`.
31
32
 
32
- **Browser exploration is tool-agnostic**: gstack (`/browse`), Playwright MCP (`browser_navigate` / `browser_snapshot` / etc.), and `openspec-pw explore --parallel N` all work — pick whichever is installed. Playwright MCP is always required (Healer uses it), so reuse it for exploration if you don't have gstack.
33
-
34
- ## Setup / Teardown
35
-
36
- Playwright supports two approaches for global lifecycle hooks. **openspec-playwright uses project dependencies** (the recommended approach) for full feature support.
37
-
38
- ### Comparison
39
-
40
- | Feature | Project Dependencies | globalSetup/globalTeardown |
41
- |---------|---------------------|---------------------------|
42
- | HTML report visibility | ✅ Shown as project | ❌ Not shown |
43
- | Trace recording | ✅ Full support | ❌ Not supported |
44
- | Playwright fixtures | ✅ Fully supported | ❌ Not supported |
45
- | Browser via fixture | ✅ Automatic | ❌ Manual launch |
46
-
47
- ### Current Implementation
48
-
49
- **Setup project** (enabled by default):
50
- - `tests/playwright/auth.setup.ts` — authenticates once, saves session to `./playwright/.auth/user.json`
51
- - All test projects depend on setup via `dependencies: ['setup']`
52
-
53
- **Teardown project** (optional, disabled by default):
54
- - `tests/playwright/global.teardown.ts` — runs AFTER all tests complete
55
- - Use for: database cleanup, uploaded file removal, cache invalidation
56
- - Enable by uncommenting in `playwright.config.ts`
57
-
58
- ### When to Enable Teardown
59
-
60
- Enable teardown when your tests create persistent data that should be cleaned up:
61
-
62
- | Scenario | Action |
63
- |----------|--------|
64
- | Tests create database records | ✅ Enable teardown, add DB cleanup |
65
- | Tests upload files | ✅ Enable teardown, add file cleanup |
66
- | Tests only read data | ❌ No teardown needed |
67
- | Tests use ephemeral/isolated environments | ❌ No teardown needed |
68
-
69
- ### Enabling Teardown
70
-
71
- 1. Copy template: `cp templates/global.teardown.ts tests/playwright/global.teardown.ts`
72
- 2. Customize cleanup logic in the file
73
- 3. Uncomment in `playwright.config.ts`:
74
- ```typescript
75
- projects: [
76
- { name: 'setup', testMatch: /.*\.setup\.ts/ },
77
- { name: 'teardown', testMatch: /global\.teardown\.ts/ }, // Uncomment
78
- {
79
- name: 'chromium',
80
- // ...
81
- teardown: 'teardown', // Uncomment
82
- },
83
- ],
84
- ```
85
-
86
- ## Testing principles
87
-
88
- **UI first** — Test every user flow through the browser UI. E2E validates that users can accomplish tasks in the real interface, not just that the backend responds correctly.
89
-
90
- ```
91
- 用户操作 → 浏览器 UI → 后端 → 数据库 → UI 反馈
92
33
  ```
93
-
94
- **API only as fallback** See **Mock data rule** below. `page.request` is acceptable for pre-condition setup (preparing test data) and API-level mocking via `page.route()`. Every **final assertion** about visible UI state must use UI selectors — never use `page.request` to assert something the user can see on screen.
95
-
96
- **Decision rule (per assertion)**:
97
-
98
- ```
99
- Can the user SEE this on screen?
100
- → Yes → MUST use: page.getByRole/ByLabel/ByText + expect()
34
+ Can user SEE this on screen?
35
+ Yes MUST use: UI selector + expect()
101
36
  → No → Record reason → page.request acceptable
102
37
  ```
103
38
 
104
- **Business logic assertion rule — numerical/calculated values MUST use API assertion**:
105
-
106
- UI assertions verify **rendering** correctness, not **calculation** correctness. A UI that correctly displays a wrong value will pass UI-only tests.
107
-
108
- ```
109
- Is the assertion about a computed/counted/calculated value?
110
- (e.g., balance, total, discount, count, percentage, score, ranking)
111
- → Yes → Use page.request to fetch backend data → assert the raw value
112
- → No → UI assertion is sufficient
113
- ```
114
-
115
- **Mock data rule:**
116
-
117
- - **Frontend: forbidden.** UI interactions must use real browser + real app data. Mocking frontend code (JS variables, component state, module-level stubs) hides real integration issues. If the frontend cannot reach a scenario through normal UI flow → **ask the user**.
118
- - **API: allowed at HTTP level.** Use `page.route()` to intercept and mock API responses (status codes, body data, latency) when:
119
- - Triggering HTTP 5xx/4xx error responses (hard to reach via UI)
120
- - Edge cases requiring pre-condition data that UI cannot set up
121
- - Third-party API failures (payment, SMS, email providers)
122
- - **Scope: API level only** — `page.route()` intercepts HTTP; do NOT mock at the database or backend service level. Mocking below the HTTP layer bypasses the real API contract and produces false confidence.
123
- - **User consent required.** Before using `page.route()` mocking, stop and ask:
124
- ```
125
- API mocking needed for: <reason>
126
- Mocked endpoint: <URL or pattern>
127
- Expected behavior: <what the test verifies>
128
- Reply **yes** to proceed, or tell me to find a UI-based approach instead.
129
- ```
130
- If the user says no → attempt a UI-based approach or skip that test case.
131
-
132
- **Examples where API assertion is required:
133
-
134
- ```typescript
135
- // ❌ UI-only assertion — hides calculation bugs
136
- await page.getByText('¥800').click(); // buy item
137
- await expect(page.getByText('总金额: ¥800')).toBeVisible(); // passes even if backend rounded wrong
39
+ **Business logic assertion rule**: computed/counted values (balance, total, count, percentage) MUST use API assertion. UI assertions verify rendering, not calculation.
138
40
 
139
- // API assertioncatches calculation bugs
140
- const order = await page.request.get(`${BASE_URL}/api/orders/${orderId}`);
141
- const body = await order.json();
142
- expect(body.total).toBe(800); // backend calculation is verified
41
+ **Mock data rule**: Frontend mocking forbidden (no JS stubs, module stubs). API mocking via `page.route()` allowed for 4xx/5xx, edge cases, third-party failures with user consent and at HTTP level only. Never mock below HTTP (DB, backend service).
143
42
 
144
- // UI-only assertion hides optimistic update failures
145
- await page.getByRole('button', { name: '点赞' }).click();
146
- await expect(page.getByText('1 个赞')).toBeVisible(); // passes even if POST failed silently
43
+ **API assertion examples**: computed/counted values MUST use `page.request` to verify, not UI:
147
44
 
148
- // ✅ API assertion — catches backend sync failures
149
- const post = await page.request.get(`${BASE_URL}/api/posts/${postId}`);
150
- const body = await post.json();
151
- expect(body.likeCount).toBeGreaterThan(0); // backend state is verified
45
+ ```
46
+ // API assertion catches calculation bugs
47
+ const order = await page.request.get(`${BASE_URL}/api/orders/${id}`);
48
+ expect((await order.json()).total).toBe(800);
152
49
 
153
50
  // ✅ Optimistic update with API verification
154
- await page.getByRole('button', { name: '点赞' }).click();
155
- await expect(page.getByText('1 个赞')).toBeVisible(); // optimistic UI
156
- await page.waitForResponse(r => r.url().includes('/api/like')); // wait for backend
157
- const post = await page.request.get(`${BASE_URL}/api/posts/${postId}`);
158
- expect((await post.json()).likeCount).toBeGreaterThan(0); // verify persistence
51
+ await expect(page.getByText('总金额: ¥800')).toBeVisible();
52
+ await page.waitForResponse(r => r.url().includes('/api/like'));
159
53
  ```
160
54
 
161
- **Never use API calls to replace routine UI flows.** If a test completes in < 200ms, it is almost certainly using `page.request` instead of real UI interactions.
55
+ **Never use API to replace routine UI flow.** If a test completes in <200ms, it skips real UI.
162
56
 
163
57
  ## Steps
164
58
 
165
59
  ### 1. Select the change or mode
166
60
 
167
- **Change mode** (`/opsx:e2e <name>`):
168
-
169
- - Use provided name, or infer from context, or auto-select if only one exists
170
- - If ambiguous `openspec list --json` + AskUserQuestion
171
- - Verify specs exist: `openspec status --change "<name>" --json`
172
- - If specs empty **STOP: E2E requires specs.** Use "all" mode instead.
173
-
174
- **"all" mode** (`/opsx:e2e all` — no OpenSpec needed):
175
-
176
- - Announce: "Mode: full app exploration + Page Object discovery"
177
- - **Goal**: Discover new routes, extract selectors, and build `pages/*.ts` Page Objects — accumulated asset for future Change tests
178
- - **Route discovery** (in order):
179
- 1. **sitemap.xml**: navigate to `${BASE_URL}/sitemap.xml` → parse URLs
180
- 2. **Link extraction**: Navigate to `${BASE_URL}/` → evaluate JS to extract all `<a href>`:
181
- ```javascript
182
- // Extract all internal links from current page
183
- () => {
184
- const origin = window.location.origin;
185
- const links = Array.from(document.querySelectorAll('a[href]'));
186
- return links
187
- .map(a => a.href)
188
- .filter(h => h.startsWith(origin) && !h.includes('/logout') && !h.includes('/api/'))
189
- .map(h => new URL(h).pathname);
190
- }
191
- ```
192
- 3. **Fallback common paths**: `/`, `/login`, `/dashboard`, `/admin`, `/profile`, `/settings`
193
-
194
- **Decision table — route discovery fallback:**
61
+ **Change mode**: Use provided name, infer from context, or auto-select if only one exists. If ambiguous → `openspec list --json` + AskUserQuestion. If specs empty → STOP, suggest "all" mode.
62
+
63
+ **"all" mode**: Route discovery priority:
64
+ 1. sitemap.xml (navigate to `${BASE_URL}/sitemap.xml` parse URLs)
65
+ 2. Link extraction (navigate home extract `<a href>` for internal paths)
66
+ 3. Fallback common paths (`/`, `/login`, `/dashboard`, `/admin`, `/profile`, `/settings`)
195
67
 
196
68
  | Situation | Action |
197
- | --- | --- |
198
- | `sitemap.xml` returns 200 with URLs | Parse all URLs → extract pathname |
199
- | `sitemap.xml` returns 404/5xx | Skip → use link extraction |
200
- | Link extraction finds 0 links | Fall back to common paths |
201
- | Common path returns 200 | Add to routes |
202
- | Duplicate routes from multiple sources | Deduplicate by pathname |
69
+ | --------- | ------ |
70
+ | sitemap.xml returns 200 with URLs | Parse → extract pathname |
71
+ | sitemap.xml 404/5xx | Skip → link extraction |
72
+ | Link extraction 0 links | Fallback common paths |
73
+ | Duplicates | Deduplicate by pathname |
203
74
 
204
- - **Persist routes**: Write discovered routes to `app-knowledge.md` → **Routes** table. Replace the entire table (including header) with fresh data do not append.
205
- - Group routes: Guest vs Protected (by attempting direct access)
75
+ Persist routes to `app-knowledge.md` → **Routes** table (replace entire table, do not append). Group routes: Guest vs Protected (by direct access attempt).
206
76
 
207
77
  ### 2. Detect auth
208
78
 
209
- **Change mode**: Read specs and extract functional requirements. Detect auth from keywords.
210
-
211
- **"all" mode**: Detect auth by attempting to access known protected paths (e.g. `/dashboard`, `/profile`). If redirected to `/login` → auth required.
212
-
213
- **Auth detection — both modes** (BOTH conditions required):
214
-
215
- **Condition A — Explicit markers**: "login", "signin", "logout", "authenticate", "protected", "authenticated", "session", "unauthorized", "jwt", "token", "refresh", "middleware"
79
+ **Change mode**: Read specs, detect auth from keywords. **"all" mode**: Try accessing protected paths → redirected to `/login` → auth required.
216
80
 
217
- **Condition B — Context indicators**: Protected routes ("/dashboard", "/profile", "/admin"), role mentions ("admin", "user"), redirect flows
218
-
219
- **Exclude false positives**: HTTP header examples (`Authorization: Bearer ...`) and code snippets do not count.
220
-
221
- **Confidence — decision table:**
81
+ **Both conditions required**:
82
+ - **A — Explicit markers**: "login", "signin", "logout", "authenticate", "protected", "session", "unauthorized", "jwt", "token", "refresh", "middleware"
83
+ - **B Context indicators**: Protected routes, role mentions ("admin", "user"), redirect flows
222
84
 
223
85
  | Confidence | Condition | Action |
224
- | --- | --- | --- |
225
- | High | Multiple markers AND context indicators | Auto-proceed |
226
- | Medium | Single marker, context unclear | Proceed + note in output |
227
- | Low | No markers found | Skip auth, test as guest |
86
+ | ---------- | --------- | ------ |
87
+ | High | Multiple markers + context indicators | Auto-proceed |
88
+ | Medium | Single marker, context unclear | Proceed + note |
89
+ | Low | No markers | Skip auth, test as guest |
228
90
 
229
- ### 3. Validate environment
91
+ Exclude false positives: HTTP header examples and code snippets do not count.
230
92
 
231
- Run the seed test before generating tests:
93
+ ### 3. Validate environment
232
94
 
233
95
  ```bash
234
96
  npx playwright test tests/playwright/seed.spec.ts --project=chromium
235
97
  ```
236
98
 
237
- This targets a single file with a specific project — it does NOT run the full suite (see note above). Seed test initializes the `page` context — it runs all fixtures, hooks, and globalSetup. Not just a smoke check: it also validates that auth setup, BASE_URL, and Playwright are fully functional.
238
-
239
- **If seed test fails**: Stop and report. Fix the environment before proceeding.
99
+ If seed test fails STOP. Fix environment before proceeding. This validates BASE_URL, auth setup, and Playwright are functional.
240
100
 
241
101
  ### 4. Explore application
242
102
 
243
- **Prerequisites**:
244
- 1. At least one browser exploration tool installed (gstack / Playwright MCP / `openspec-pw explore`).
245
- 2. seed test pass
246
- 3. BASE_URL must be verified reachable (see 4.1)
247
-
248
- If auth is required and `auth.setup.ts` already exists → auth is ready. If auth is not yet configured → use the workaround below (Option B for protected routes).
249
-
250
- #### 4.1. Verify BASE_URL + Read app-knowledge.md
251
-
252
- 1. **Verify BASE_URL**: navigate to `<BASE_URL>` → if HTTP 5xx → **STOP: backend error. Fix app first.**
253
- 2. **Read app-knowledge.md**: known risks, project conventions
254
- 3. **Routes** (from Step 1): use already-discovered routes — no need to re-extract
255
-
256
- #### 4.2. Explore each route
257
-
258
- For each route: navigate → check console for errors → snapshot DOM → screenshot.
259
-
260
- #### Alternative: parallel exploration (via `openspec-pw explore`)
261
-
262
- If you have ≥5 routes, skip 4.2 and use the dedicated CLI for genuine parallel exploration:
263
-
264
- ```bash
265
- openspec-pw explore --parallel 4 # 4 independent Chromium workers
266
- openspec-pw explore --dry-run # preview chunk assignment first
267
- ```
268
-
269
- **Why**: a single shared browser instance with one active page causes `Promise.allSettled` on navigation/snapshot commands to serialize execution and create navigation state conflicts. `openspec-pw explore` launches N independent browser processes, each with its own Chromium context, for genuine parallelism.
103
+ **Prerequisites**: browser tool installed, seed test passed, BASE_URL reachable.
270
104
 
271
- **After parallel exploration completes**, read the updated `app-exploration.md` to continue with Step 4.3.
105
+ **4.1. Verify BASE_URL**: navigate if HTTP 5xx → **STOP: backend error**. Read `app-knowledge.md` for known risks and conventions.
272
106
 
273
- **After navigating, check for app-level errors**:
274
-
275
- | Signal | Meaning | Action |
276
- | ----------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------- |
277
- | HTTP 5xx or unreachable | Backend/server error | **STOP** — tell user: "App has a backend error (HTTP <code>). Fix it, then re-run `/opsx:e2e <name>` to re-explore." |
278
- | JS error in console | App runtime error | **STOP** — tell user: "Page has JS errors. Fix them, then re-run `/opsx:e2e <name>` to re-explore." |
279
- | HTTP 404 | Route not in app (metadata issue) | Continue — mark `⚠️ route not found` in app-exploration.md |
280
- | Auth required, no credentials | Missing auth setup | Continue — skip protected routes, explore login page |
281
- | Suspicious network request | API returned 4xx/5xx | Continue — mark `⚠️ API error: <endpoint> returned <code>` in app-exploration.md |
282
-
283
- **Redirect / Refresh loop detection** — run after initial navigate:
284
-
285
- ```
286
- // 1. Initial capture
287
- navigate <url>
288
- wait for network-idle // wait for SPA hydration
289
- evaluate "window.location.href" // url1
290
- wait 2000ms
291
-
292
- // 2. Observe stability
293
- evaluate "window.location.href" // url2
294
- check console for errors
295
-
296
- // 3. Detect
297
- if (url1 !== url2) {
298
- → ❌ URL changed — redirect loop
299
- → Is this route protected without valid auth?
300
- → Yes → auth.setup.ts is broken → fix auth first
301
- → No → App middleware bug → mark route ❌ skip, record as App Bug
302
- }
303
- if (console errors/warnings > 10) {
304
- → ❌ Excessive console errors — page refresh / JS crash loop
305
- → Mark route ❌ skip, record as App Bug
306
- }
307
- ```
308
-
309
- **Network monitoring**: After navigating, check network requests for failed API calls. Failed requests (status ≥ 400) on a route indicate an API/backend issue — record in `app-exploration.md` for reference.
310
-
311
- **For guest routes** (no auth):
312
-
313
- ```
314
- navigate <url>
315
- ```
316
-
317
- **For protected routes** (auth required):
318
-
319
- ```
320
- // Option A: use existing storageState (recommended)
321
- // Option B: navigate to /login first, fill form, then navigate to target
322
- // Option C: set auth cookies directly via document.cookie
323
- ```
107
+ **4.2. Explore each route**: navigate → check console errors → snapshot DOM → screenshot. For ≥5 routes, use `openspec-pw explore --parallel N` for genuine parallel browsers.
324
108
 
325
- **If credentials are not yet available**:
109
+ **App-level error decisions**:
326
110
 
327
- 1. Skip protected routes — mark `⚠️ auth needed — explore after auth.setup.ts`
328
- 2. Explore the login page itself (guest route) — extract form selectors
329
- 3. After auth.setup.ts runs, re-run exploration for protected routes
111
+ | Signal | Action |
112
+ | ------ | ------ |
113
+ | HTTP 5xx / unreachable | **STOP** — backend error, fix app first |
114
+ | JS error in console | **STOP** — page has JS errors |
115
+ | HTTP 404 | Continue — mark `⚠️ route not found` |
116
+ | Auth required, no credentials | Continue — skip protected routes, explore login page |
117
+ | API 4xx/5xx | Continue — mark `⚠️ API error` |
330
118
 
331
- Wait for page stability:
119
+ **Redirect/Refresh loop detection**: Navigate → wait networkidle → capture URL → wait 2s → capture URL again. If URL changed → redirect loop (auth issue or middleware bug). If console errors > 10 → refresh loop. Skip route, record as App Bug.
332
120
 
333
- - **React 19 / Next.js App Router**: use `page.waitForLoadState('networkidle')` React 19 concurrent mode batches events asynchronously; 200-500ms timeouts are unreliable under resource contention
334
- - **Vue 2/3 / Angular / React 18 / Plain JS / jQuery**: `waitForSelector(targetElement)` is sufficient and faster — DOM updates are synchronous; Playwright's actionability checks auto-wait correctly
335
- - Prefer specific element waits (`waitForSelector`) over generic load states
336
- - Ready signal: heading, spinner disappears, or URL change
121
+ **Page wait stability**: React 19 / Next.js App Router `page.waitForLoadState('networkidle')` (concurrent mode batches async). Vue/Angular/React 18/plain JS `waitForSelector(targetElement)` (DOM updates synchronous). Prefer specific element waits over generic load states.
337
122
 
338
- #### 4.3. Parse the snapshot
123
+ **Element extraction from snapshot**:
339
124
 
340
- From the DOM snapshot output, extract **interactive elements** for each route:
125
+ | Element type | Selector priority |
126
+ | ------------ | ----------------- |
127
+ | Buttons | `[data-testid]` > `getByRole` > `getByLabel` > `getByText` |
128
+ | Form fields | `[data-testid]` > `name` > `label` |
129
+ | Navigation | `text` > `href` |
130
+ | Headings, errors | For assertions |
131
+ | Special (canvas, iframe, CAPTCHA, OTP, Shadow DOM, file upload) | Detect & note strategy |
341
132
 
342
- | Element type | What to capture | Selector priority |
343
- | -------------------- | ------------------------------------ | ---------------------------------------------------------- |
344
- | **Buttons** | text, selector | `[data-testid]` > `getByRole` > `getByLabel` > `getByText` |
345
- | **Form fields** | name, type, label, selector | `[data-testid]` > `name` > `label` |
346
- | **Navigation links** | text, href, selector | `text` > `href` |
347
- | **Headings** | text content, selector | for assertions |
348
- | **Error messages** | text patterns, selector | for error path testing |
349
- | **Dynamic content** | structure — row counts, card layouts | for data-driven tests |
350
- | **Special elements** | type, selector, dimensions | for canvas/iframe/Shadow DOM test strategies |
133
+ **Special elements quick reference**:
351
134
 
352
- #### 4.3.1. Detect special elements
135
+ | Element | Snapshot signal | Strategy |
136
+ | ------- | --------------- | -------- |
137
+ | `<canvas>` | role="img" | `evaluate: getContext`, boundingBox > 0 |
138
+ | `<iframe>` | role="iframe" | frameLocator + src attr |
139
+ | CAPTCHA | `.g-recaptcha`, `[data-sitekey]` | auth.setup bypass / test.skip |
140
+ | OTP | 6-digit input fields, maxLength=1 | Dev bypass / E2E_OTP_CODE |
141
+ | Shadow DOM | role="generic" no children | `evaluate: el.shadowRoot` |
142
+ | Rich text | `[contenteditable]` | type + textContent |
143
+ | File upload | `<input type="file">` | setInputFiles |
353
144
 
354
- From the DOM snapshot and JS evaluation, identify these special elements per route:
145
+ Record findings in `app-exploration.md`. Output path:
146
+ - Change mode: `openspec/changes/<name>/specs/playwright/app-exploration.md`
147
+ - All mode: `<root>/app-exploration.md`
355
148
 
356
- **Special element detection matrix:**
149
+ **Idempotency**: If `app-exploration.md` exists → read, verify routes, update only changed/new routes. **Route Snapshot Hash**: Navigate to sitemap.xml → hash content → if unchanged since last exploration, skip re-exploration entirely. Store hash in `app-knowledge.md` → `Exploration State`.
357
150
 
358
- | Element | Snapshot signal | Evaluate supplement | Exploration priority |
359
- | ------- | --------------- | ---------------------------------------------- | ------------------- |
360
- | `<canvas>` | `role="img"`, `tagName="CANVAS"` | `canvas.getContext('2d'/'webgl')`, `width`, `height` | High |
361
- | `<iframe>` | `role="iframe"`, `src` attribute | `frameLocator` available | High |
362
- | CAPTCHA | `.g-recaptcha`, `.h-captcha`, `[data-sitekey]`, canvas+slider | recaptcha score via API (if configured) | High |
363
- | OTP / SMS | 6-digit input, countdown timer | Check if dev bypass exists | High |
364
- | Shadow DOM | `role="generic"` with no children | Check `shadowRoot` via evaluate | Medium |
365
- | Rich text editor | `[contenteditable]`, `role="textbox"` | `innerHTML`, `getContent()` | Medium |
366
- | Video / Audio | `role="application"` or name contains "video"/"audio" | `evaluate` checks both `<video>` and `<audio>` tags | Medium |
367
- | File upload | `<input type="file">` | `accept` attribute, `multiple` flag | Medium |
368
- | Drag-and-drop | drag events in JS | Simulate DnD via coordinate clicks | Low |
369
- | Date picker | specific `data-testid` or class patterns | Click triggers → evaluate value | Low (skip unless specs mention) |
370
- | Infinite scroll | Dynamic row insertion | Count elements before/after scroll | Low (skip unless specs mention dynamic lists/pagination) |
371
- | WebSocket / SSE | No DOM signal | Check page console for WS events | Low (check only if app uses real-time features) |
151
+ **4.3. Update shared knowledge**: Extract project-level findings to `tests/playwright/app-knowledge.md`. Auto-de-duplicate by key.
372
152
 
373
- **For each detected special element, capture via JS evaluation with targeted DOM queries:**
374
- - Canvas: `getContext('webgl2'/'webgl'/'2d')`, `width`, `height`
375
- - Iframe: `src` attribute → use `frameLocator` in tests
376
- - CAPTCHA: `.g-recaptcha`, `.h-captcha`, `[data-sitekey]`, canvas+slider detection
377
- - OTP: `input` elements with `maxLength === 1` or `type === 'tel'`
378
- - Rich text: `[contenteditable]` → `innerHTML`, `textContent.length`
379
- - Video/Audio: `querySelector('video'/'audio')` → `paused`, `duration`
380
- - Shadow DOM: `role="generic"` with no children → check `shadowRoot`
153
+ ### 5. Generate test plan (change mode only)
381
154
 
382
- Record findings in `app-exploration.md` **Special Elements Detected** table.
155
+ **All mode**: skip show confirmation, proceed to Step 6.
383
156
 
384
- #### 4.4. Write app-exploration.md
157
+ **Prerequisite**: `app-exploration.md` must exist → STOP and run Step 4 if missing.
385
158
 
386
- Output: `openspec/changes/<name>/specs/playwright/app-exploration.md`
159
+ Create `openspec/changes/<name>/specs/playwright/test-plan.md`. Read inputs: specs, app-exploration.md, app-knowledge.md. Create test cases (functional requirement → test case, with `@role` and `@auth` tags). Reference verified selectors from exploration.
387
160
 
388
- **"all" mode** (no change context): Output to `<root>/app-exploration.md` instead. The `<root>/` version also serves as the INPUT file read by `openspec-pw explore` (which updates only the Status column).
161
+ **State mutual exclusion**: before each test case, identify state boundaries and which elements disappear/appear. Assert mutual exclusion explicitly.
389
162
 
390
- Key fields per route:
163
+ **Idempotency**: If test-plan.md exists → read and supplement missing cases, never regenerate.
391
164
 
392
- - **URL**: `${BASE_URL}<path>`
393
- - **Auth**: none / required (storageState: `<path>`)
394
- - **Ready signal**: how to know the page is loaded
395
- - **Elements**: interactive elements with verified selectors (see 4.3 table)
396
- - **Screenshot**: `__screenshots__/<slug>.png`
397
-
398
- After exploration, add route-level notes (redirects, dynamic content → see 4.5).
399
-
400
- #### 4.5. Exploration behavior notes
401
-
402
- | Situation | Action |
403
- | ------------------------------------------------- | ---------------------------------------------------------------- |
404
- | SPA routing (URL changes but page doesn't reload) | Explore via navigation clicks from known routes, not direct URLs |
405
- | Page loads but no interactive elements | Wait longer for SPA hydration |
406
- | Dynamic content (user-specific) | Record structure — use `toContainText` or regex, not `toHaveText` |
407
-
408
- **Idempotency**: If `app-exploration.md` already exists → read it, verify routes still match the live app, update only new routes or changed pages.
409
-
410
- #### Route Snapshot Hash -- Skip Unchanged Routes
411
-
412
- Before re-exploring, compute a lightweight hash of the app's current state:
413
-
414
- 1. **Quick hash** (for re-runs): Navigate to `${BASE_URL}/sitemap.xml` → hash the XML content
415
- 2. **If hash unchanged** since last exploration: Skip re-exploration entirely -- use cached `app-exploration.md`
416
- 3. **If hash changed**: Re-explore only changed routes (diff sitemap XML)
417
- 4. **Store hash** in `app-knowledge.md` → `Exploration State` section
418
-
419
- ```markdown
420
- ## Exploration State
421
- Last explored: 2026-04-12T07:30:00Z
422
- Sitemap hash: sha256:abc123...
423
- Routes count: 20
424
- ```
425
-
426
- This prevents redundant exploration on every `/opsx:e2e` run.
427
-
428
- #### 4.6. Update app-knowledge.md
429
-
430
- After writing `app-exploration.md`, extract **project-level shared knowledge** and append to `tests/playwright/app-knowledge.md`. **Auto-de-duplicate**: before adding any row, check if the same key already exists — if so, skip.
431
-
432
- | Section | What to extract | De-duplication key |
433
- | ------- | --------------- | ----------------- |
434
- | Architecture | Monolith or separated? Backend port? Restart command? | **Section-level**: update existing rows instead of adding duplicates |
435
- | Credential Format | Login endpoint, username format (email vs username) | **Field-level**: `Field` column (username, password, login endpoint) |
436
- | Common Selector Patterns | New patterns discovered that apply across routes | **Element + Selector**: skip if same Element + Selector already exists |
437
- | SPA Routing | SPA framework, routing behavior | **Section-level**: update existing instead of appending |
438
- | Project Conventions | BASE_URL, auth method, multi-user roles | **Convention column**: skip if same convention already exists |
439
- | Selector Fixes | Healed selectors (see Step 9 Phase 2-7) — route, old → new selector, reason, date | **Route + Old Selector**: skip if same key exists |
440
- | Assertion Fixes | Healed assertions (see Step 9 Phase 2-7) — test, old → new assertion, reason, date | **Test + Old Assertion**: skip if same key exists |
441
-
442
- Append only new/changed items — preserve existing content.
443
-
444
- ### 4.5. Vision Check (Optional, VLM-powered)
445
-
446
- If Ollama with a vision model is available, analyze screenshots for layout anomalies before proceeding to test generation. This catches UI issues early — before tests fail.
447
-
448
- **Prerequisite**: Run `openspec-pw doctor` to check Vision Check availability:
449
- ```
450
- ─── Vision Check ───
451
- ✓ ollama: http://localhost:11434 (qwen2.5-vl)
452
- ```
453
-
454
- If Vision Check shows `⚠ disabled` or `⚠ not reachable` → skip this step entirely. Vision Check is optional and should not block exploration.
455
-
456
- **Configuration**:
457
- - `.env` file in `tests/playwright/` (highest priority): `OLLAMA_URL`, `OLLAMA_VISION_MODEL`, `OLLAMA_VISION_ENABLED`
458
- - Environment variables: `OLLAMA_URL`, `OLLAMA_VISION_MODEL`
459
-
460
- If no configuration is found, vision check is disabled.
461
-
462
- **Run vision check** after exploration is complete:
463
-
464
- ```bash
465
- openspec-pw vision-check --screenshots "__screenshots__/*.png"
466
- ```
467
-
468
- **Flags**:
469
- - `--screenshots <glob>` — Screenshot paths (required)
470
- - `--parallel <n>` — Concurrent Ollama requests (default: 4)
471
- - `--severity <levels>` — Filter by severity: `blocking,warning,minor`
472
- - `--output <path>` — Write JSON results to file
473
- - `--dry-run` — List screenshots without analyzing
474
- - `--json` — Output JSON format
475
-
476
- **Exit codes**:
477
- - `0` = Check completed (with or without anomalies)
478
- - `1` = Ollama not available → skip vision check, continue workflow
479
- - `2` = Configuration disabled → skip vision check, continue workflow
480
-
481
- **Anomaly types detected**:
482
-
483
- | Type | Description | Example |
484
- | --- | --- | --- |
485
- | `obscured` | Interactive element covered by another element | Submit button hidden behind modal |
486
- | `crowded` | Elements too close together, hard to distinguish | Form fields with <8px spacing |
487
- | `overflowed` | Content clipped by container | Truncated text without ellipsis |
488
-
489
- **Output handling**:
490
-
491
- 1. **JSON output** (with `--json`): Parse anomalies array for programmatic use
492
- 2. **Auto-append**: Without `--json`, anomalies are appended to `app-exploration.md` → **Visual Anomalies** table
493
- 3. **Decision table**:
494
-
495
- | Anomaly Severity | Action |
496
- | --- | --- |
497
- | `blocking` | Flag route with `⚠️ Layout`, consider skipping test generation until fixed |
498
- | `warning` | Log to Visual Anomalies table, continue test generation |
499
- | `minor` | Log for reference, no action required |
500
-
501
- **Graceful degradation**: If Ollama is unavailable or times out → log warning, skip vision check, continue to Step 5. Vision check should never block the E2E workflow.
502
-
503
- #### 4.7. After exploration
504
-
505
- Pass `app-exploration.md` to:
506
-
507
- - **Step 5 (Planner)**: reference real routes, auth states, and elements in test-plan.md
508
- - **Step 6 (Generator)**: use verified selectors instead of inferring
509
-
510
- Read `tests/playwright/app-knowledge.md` as context for cross-change patterns.
511
-
512
- ### 5. Generate test plan
513
-
514
- > **"all" mode: skip test-plan generation.** No OpenSpec specs → no test-plan to generate. Still show confirmation below, then proceed to Step 6.
515
-
516
- **All mode — brief confirmation before Step 6:**
517
- ```
518
- ## All Mode: Page Object Discovery
519
- Discovered <N> routes (<M> guest, <K> protected)
520
- Special elements: <element summary>
521
- Ready to generate Page Objects for: <page-name>Page.ts, <page-name>Page.ts, ...
522
- Reply **yes** to proceed, or tell me to exclude routes or adjust strategies.
523
- ```
524
-
525
- **Change mode — prerequisite**: If `openspec/changes/<name>/specs/playwright/app-exploration.md` does not exist → **STOP**. Run Step 4 (explore application) before generating tests. Without real DOM data from exploration, selectors are guesses and tests will be fragile. **All mode**: if `<root>/app-exploration.md` does not exist → same STOP.
526
-
527
- **Change mode**: Create `openspec/changes/<name>/specs/playwright/test-plan.md`.
528
-
529
- **Read inputs**: specs, app-exploration.md, app-knowledge.md
530
-
531
- **Create test cases**: functional requirement → test case, with `@role` and `@auth` tags. Reference verified selectors from app-exploration.md.
532
-
533
- > **State mutual exclusion**: before creating each test case — where are the state boundaries? During state transitions, which elements disappear or appear? Mutual exclusion must be asserted explicitly.
534
-
535
- If a test case requires `page.route()` API mocking → append `⚠️ API Mock` flag to the test case line in the summary, with reason.
536
-
537
- **Idempotency**: If test-plan.md exists → read and use, **but you MAY supplement missing test cases**. "Do not regenerate" means: do not discard existing cases, but you CAN add new ones discovered during Step 4 exploration that weren't in the original spec (e.g., empty states, error paths found during DOM exploration).
538
-
539
- **⚠️ Human verification — STOP before generating code.**
540
-
541
- After creating (or reading existing) test-plan.md, **stop and display the test plan summary** for user confirmation:
542
-
543
- **Output format** — show the test plan in markdown directly in the conversation:
544
-
545
- ````markdown
546
- ## Test Plan Summary: `<change-name>`
547
-
548
- **Auth**: required / not required | Roles: ...
549
-
550
- ### Test Cases
551
- - ✅ `<test-name>` — `<route>`, happy path
552
- - ✅ `<test-name>` — `<route>`, error path: `<error condition>`
553
-
554
- ### Special Elements
555
- - ⚠️ **CAPTCHA** at `<route>` — strategy: `auth.setup bypass / skip / api-only`
556
- - ⚠️ **Canvas/WebGL** at `<route>` — strategy: screenshot + dimensions
557
- - ⚠️ **OTP** at `<route>` — strategy: test credentials / dev bypass
558
- - ⚠️ **Iframe** at `<route>` — strategy: frameLocator + assert inner content
559
- - ⚠️ **Video/Audio** at `<route>` — strategy: play() + assert !paused
560
- - ⚠️ **File Upload** at `<route>` — strategy: setInputFiles + assert upload
561
- - ⚠️ **Drag-and-Drop** at `<route>` — strategy: dragAndDrop or evaluate events
562
- - ⚠️ **WebSocket/SSE** at `<route>` — strategy: waitForResponse + waitForFunction
563
-
564
- ### Not Covered
565
- - `<element or scenario not testable>`
566
- ````
567
-
568
- **Important**: Only list special elements that were actually detected in Step 4. Do not pre-populate with all possible types. If no special elements were found → omit the **Special Elements** section entirely.
569
-
570
- Then ask: "Does this coverage match your intent? Reply **yes** to proceed, or tell me what to add/change."
571
-
572
- If the user requests changes → update test-plan.md → re-display summary → re-confirm → proceed.
165
+ **⚠️ Human verification**: After creating/reading test-plan.md, **STOP** and display summary. Ask user to confirm before proceeding to Step 6.
573
166
 
574
167
  ### 6. Generate (Generator role)
575
168
 
576
- **"all" mode**: Build and expand Page Objects for future Change tests.
577
-
578
- **Prerequisite** (change mode only): If `openspec/changes/<name>/specs/playwright/app-exploration.md` does not exist → **STOP**. Run Step 4 first. For **all mode**, exploration is embedded dynamically in Step 6 — no pre-existing app-exploration.md is required.
579
-
580
- **Page Object pattern** — read before writing any page file:
169
+ **All mode**: Build Page Objects for future tests. **Change mode**: Generate `tests/playwright/changes/<name>/<name>.spec.ts`.
581
170
 
171
+ **Page Object pattern** (read BasePage.ts first):
582
172
  ```typescript
583
- // ✅ 正确:getters + async actions + this.click/fill
584
173
  export class LoginPage extends BasePage {
585
174
  get usernameInput() { return this.byLabel('用户名'); }
586
175
  get submitBtn() { return this.byRole('button', { name: '登录' }); }
@@ -591,226 +180,43 @@ export class LoginPage extends BasePage {
591
180
  await this.click(this.submitBtn);
592
181
  }
593
182
  }
594
-
595
- // ❌ 错误:测试文件里写 inline locators
596
- test('login', async ({ page }) => {
597
- await page.getByLabel('用户名').fill('user'); // ← never do this!
598
- });
599
183
  ```
600
184
 
601
- **Decision table — Page Object file handling**:
602
-
603
- | Situation | Action |
604
- | --- | --- |
605
- | `pages/<Route>Page.ts` does not exist | Create from LoginPage pattern |
606
- | File exists with some getters | Extend — add missing, preserve existing |
607
- | File exists but uses inline locators | Rewrite with Page Object pattern, keep selector strings |
608
- | Route removed from app | Remove corresponding Page Object file |
609
-
610
- **File naming**: `pages/<Route>Page.ts` — use kebab-case route → PascalCase. `/login` → `LoginPage.ts`, `/user-profile` → `UserProfilePage.ts`.
611
-
612
- For each discovered route:
613
-
614
- 1. Read existing `pages/<Route>Page.ts` (if any — incremental, not overwrite)
615
- 2. Navigate to route with correct auth state
616
- 3. browser_snapshot to extract interactive elements (see 4.3 table)
617
- 4. Write or update `pages/<Route>Page.ts` — extend with newly discovered elements
618
- 5. Also write `tests/playwright/app-all.spec.ts` — smoke test. **Minimum standard**: verify at least one heading or key interactive element is visible — not just "no crash". If the page loads but shows an empty shell or an error, the test must fail.
619
-
620
- **Output priority**: Page Objects (`pages/*.ts`) are the primary asset. Smoke test is secondary. Existing Page Objects are never overwritten — only extended.
621
-
622
- **Change mode** → `tests/playwright/changes/<name>/<name>.spec.ts` (functional):
185
+ **Page Object file handling**:
186
+ - File doesn't exist → Create
187
+ - File exists with getters → Extend (preserve existing, add missing)
188
+ - File has inline locators → Rewrite with Page Object pattern
623
189
 
624
- - Read: test-plan.md, app-exploration.md, app-knowledge.md, seed.spec.ts
625
- - For each test case: verify selectors in real browser, then write Playwright code
626
-
627
- **Per-assertion UI check** (before writing each assertion):
190
+ **Per-assertion check**:
628
191
  ```
629
192
  Is this assertion about a visible UI result?
630
193
  → Yes → MUST use: expect(locator) with page selector
631
- → No → Is this a precondition or unreachable HTTP error?
632
- → Yes → page.request is acceptable (record reason)
633
- → No → This is a bug — rewrite with UI selector
634
- ```
635
- **Never use page.request for assertions the user can see on screen.** If you wrote page.request.get() for a visible result → rewrite with expect(locator) from the browser snapshot.
636
-
637
- **Selector verification (change mode)**:
638
-
639
- 1. Navigate to route with correct auth state
640
- 2. browser_snapshot to confirm page loaded
641
- 3. For each selector: verify from current snapshot (see 4.3 table for priority)
642
- 4. Write test code with verified selectors
643
- 5. If selector unverifiable → note for Healer (Step 9)
644
-
645
- **Selector Caching — reuse Step 4 exploration results:**
646
-
647
- After Step 4, verified selectors are stored in `app-exploration.md`. Before navigating to verify a selector in Step 6, check if the route already has verified selectors from Step 4.
648
-
649
- ```javascript
650
- // Step 6: Before navigating to verify, check cached selectors
651
- const cachedSelectors = appExploration.routes.find(r => r.path === routePath)?.elements
652
- if (cachedSelectors && cachedSelectors.length > 0) {
653
- // Use first-priority selector from cache instead of re-navigating
654
- const selector = getFirstPrioritySelector(cachedSelectors)
655
- // Only navigate to verify if selector is missing or marked "Fragile"
656
- if (selector.stability !== 'Fragile') {
657
- return selector // use cached, skip verification navigation
658
- }
659
- }
660
- // Fallback: navigate + snapshot + verify
661
- ```
662
-
663
- **Benefit**: For a 50-test case suite, this saves 30-50 redundant navigations (~2-5 minutes).
664
-
665
- **File to read**: The selector caching uses data already stored in Step 4.4's `app-exploration.md` output — no new file needed.
666
-
667
- **Test coverage — empty states**: For list/detail pages, explore the empty state. If the app shows a "no data" UI when the list is empty, generate a test to verify it. Empty states are often missing from specs but are real user paths.
668
-
669
- **Test data fabrication**: never invent user accounts, product lists, or API responses. Follow §6 of the employee standards — explicitly ask the user for test data; use `TODO(user)` markers for approved placeholders; fail loudly rather than fabricate.
670
-
671
- **Test coverage — special elements**: Check `app-exploration.md` → **Special Elements Detected** table. For each special element, generate tests using the following strategies:
672
- - Canvas: screenshot + boundingBox → dimensions > 0, or 2D pixel verification
673
- - WebGL: screenshot only (no pixel comparison — rendering varies)
674
- - Iframe: `frameLocator` + assert inner content visible
675
- - Rich text: `contenteditable` → type + `textContent` assertion
676
- - Video/Audio: `play()` → assert `!paused`
677
- - CAPTCHA/OTP/File upload/Drag-drop: See AI-Opaque Elements section in templates
678
-
679
- **Test coverage — AI-opaque elements**: For CAPTCHA, OTP, slider CAPTCHA, file upload, and drag-drop — elements that Playwright cannot reliably automate:
680
-
681
- 1. Mark the element in `app-exploration.md` → **Special Elements Detected** table with type and automation strategy
682
- 2. Generate the test using the appropriate strategy:
683
- - **CAPTCHA**: Bypass via `auth.setup.ts` storageState, or skip with `test.skip()`, or verify via API
684
- - **OTP**: Use pre-verified test credentials (`E2E_OTP_CODE` env var), or development bypass flag
685
- - **File upload**: Use `page.setInputFiles()` with fixture files
686
- - **Drag-drop**: Use `page.dragAndDrop()` or `page.evaluate()` with custom event dispatching
687
- 3. If the element is truly non-automatable, write `test.skip()` with a comment explaining why, and mark with `/handoff` for manual testing
688
-
689
- **Test coverage — performance**: Generate a Core Web Vitals test **only if** the OpenSpec spec or app-exploration.md specifies explicit performance targets (e.g., "LCP must be under 2s"). If no business target is defined, skip performance testing — hard-coded thresholds (lcp < 2500ms) produce false passes and add noise.
690
-
691
- ```typescript
692
- // 🚫 Avoid for special elements:
693
- await canvas.screenshot() // screenshot alone — no dimension/size assertion
694
- await expect(canvas).toHaveScreenshot() // pixel-to-pixel comparison for WebGL
695
-
696
- // ✅ Always:
697
- const box = await canvas.boundingBox();
698
- expect(box.width).toBeGreaterThan(0);
194
+ → No → page.request acceptable (record reason)
699
195
  ```
700
196
 
701
- **Output format**:
702
-
703
- - Follow `seed.spec.ts` structure
704
- - Use `test.describe(...)` for grouping
705
- - Each test: `test('描述性名称', async ({ page }) => { ... })`
706
- - Prefer `data-testid` selectors (see 4.3 table)
197
+ **Selector caching**: Use already-verified selectors from `app-exploration.md`. Only navigate to verify if selector is missing or marked Fragile.
707
198
 
708
- #### 6.1. Use BasePage for shared navigation and selectors
199
+ **Test data fabrication**: Never invent. Follow §6 of employee standards ask user, use `TODO(user)` markers.
709
200
 
710
- Read `tests/playwright/pages/BasePage.ts` for shared utilities:
711
- - `goto(path)` — navigation with configurable `waitUntil`
712
- - `byRole(role, opts)`, `byLabel(label)`, `byPlaceholder(text)`, `byText(text)`, `byTestId(id)` — selector helpers in priority order (semantic → form → fallback)
713
- - `click(locator)`, `fill(locator, value)`, `fillAndVerify(locator, value)` — safe interactions; use `fillAndVerify` when the next action depends on the value being committed
714
- - `waitForToast(text?)`, `waitForLoad(spinnerSelector?)` — wait utilities
715
- - `reload()` — page reload with hydration
201
+ **Test coverage — empty states**: For list/detail pages, explore and test empty state UI.
716
202
 
717
- **AppPage pattern** extend BasePage for page-specific selectors:
718
-
719
- ```typescript
720
- // tests/playwright/pages/LoginPage.ts
721
- import { BasePage } from './BasePage';
722
- import type { Page } from '@playwright/test';
723
-
724
- export class LoginPage extends BasePage {
725
- get usernameInput() { return this.byLabel('用户名'); }
726
- get passwordInput() { return this.byLabel('密码'); }
727
- get submitBtn() { return this.byRole('button', { name: '登录' }); }
728
-
729
- constructor(page: Page) { super(page); }
730
-
731
- async login(user: string, pass: string) {
732
- await this.goto('/login');
733
- await this.fillAndVerify(this.usernameInput, user);
734
- await this.fillAndVerify(this.passwordInput, pass);
735
- await this.click(this.submitBtn);
736
- }
737
- }
738
- ```
739
-
740
- ```typescript
741
- // tests/playwright/changes/<name>/<name>.spec.ts
742
- import { LoginPage } from '../pages/LoginPage';
743
-
744
- test('user can login', async ({ page }) => {
745
- const loginPage = new LoginPage(page);
746
- await loginPage.login('user@example.com', 'password123');
747
- await loginPage.expectURL(/dashboard/);
748
- });
749
- ```
750
-
751
- **If a shared page object doesn't exist yet**: define it inline in the spec AND write it to `tests/playwright/pages/<PageName>.ts` so future tests can reuse it.
752
-
753
- #### 6.2. Selector patterns
754
-
755
- | Prefer (robust) | Avoid (fragile) |
756
- | --- | --- |
757
- | `getByRole`, `getByTestId`, `getByLabel` | CSS class (`'.notification-bell'`), CSS ID (`'#avatarBtn'`) |
758
- | `waitForSelector(targetElement)` | hardcoded `200ms` / `500ms` delays |
759
-
760
- See above for Page Object pattern, LoginPage example, and BasePage utilities.
761
-
762
- If the file exists → diff against test-plan, add only missing test cases.
203
+ **Performance tests**: Only if spec or app-exploration.md defines explicit targets. No hard-coded thresholds.
763
204
 
764
205
  ### 7. Configure auth (if required)
765
206
 
766
- - **API login**: Generate `auth.setup.ts` using `E2E_USERNAME`/`E2E_PASSWORD` + POST to login endpoint
767
- - **UI login**: Generate `auth.setup.ts` using browser form fill. Update selectors to match your login page
207
+ - **API login**: Generate `auth.setup.ts` using `E2E_USERNAME`/`E2E_PASSWORD` + POST
208
+ - **UI login**: Generate using browser form fill
768
209
  - **Multi-user**: Separate `storageState` paths per role
769
210
 
770
- **Credential format guidance**:
211
+ Always use env vars, never hardcode. If auth.setup.ts exists → verify, update only if stale.
771
212
 
772
- - If the app uses **email** for login use `CHANGE_ME@example.com`
773
- - If the app uses **username** (alphanumeric + underscore) → use `test_user_001` (more universal)
774
- - Check existing test files or login page to determine the format
775
- - Always set credentials via environment variables — never hardcode
776
-
777
- **Prompt user**:
778
-
779
- ```
780
- Auth required. To set up:
781
- 1. Customize tests/playwright/credentials.yaml
782
- 2. Export: export E2E_USERNAME=xxx E2E_PASSWORD=yyy
783
- 3. Run auth: npx playwright test --project=setup
784
- 4. Then run tests: openspec-pw run <name> # skips to Step 9 directly (artifacts are reused)
785
- ```
786
-
787
- **Idempotency**: If `auth.setup.ts` already exists → verify format, update only if stale.
788
-
789
- **Post-auth re-exploration**: If Step 4 skipped protected routes due to missing auth, re-run exploration for those routes now that auth is configured. Navigate to each protected route with auth context → snapshot DOM → update `app-exploration.md`. Selectors verified now are better than guesses used during test generation.
213
+ **Post-auth re-exploration**: If Step 4 skipped protected routes, re-run exploration for those routes now.
790
214
 
791
215
  ### 8. Configure playwright.config.ts
792
216
 
793
- **Output**: `playwright.config.ts` (project root; or `tests/playwright/playwright.config.ts` if config already exists there)
794
-
795
- If missing → generate a minimal `playwright.config.ts` with webServer, projects, and reporters.
796
-
797
- **Auto-detect BASE_URL** (in priority order):
798
-
799
- 1. `process.env.BASE_URL` if already set
800
- 2. `tests/playwright/seed.spec.ts` → extract `BASE_URL` value
801
- 3. Read `vite.config.ts` (or `vite.config.js`) → extract `server.port` + infer protocol (`https` if `server.https`, else `http`)
802
- 4. Read `package.json` → `scripts.dev` or `scripts.start` → extract port from `--port` flag
803
- 5. **Python projects**: Read `pyproject.toml` or `settings.py` → extract `PORT` or `DEBUG` config
804
- 6. **Go projects**: Read `main.go` or `.env` → extract port from `http.ListenAndServe` or env vars
805
- 7. Fallback: `http://localhost:3000`
217
+ If missing generate minimal config with webServer, projects, reporters. If exists → preserve all fields, add only missing webServer block.
806
218
 
807
- **Auto-detect dev command**:
808
-
809
- 1. `package.json` → scripts in order: `dev` → `start` → `serve` → `preview` → `npm run dev`
810
- 2. **Python projects**: Check for `uvicorn`, `flask run`, `python manage.py runserver`, `fastapi dev`
811
- 3. **Go projects**: Check for `air`, `reflex`, `fresh`, or custom dev scripts
812
-
813
- If playwright.config.ts exists → READ first, preserve ALL existing fields, add only missing `webServer` block.
219
+ **BASE_URL auto-detect**: process.env.BASE_URL → seed.spec.ts → vite.config.ts → package.json scripts → fallback `http://localhost:3000`.
814
220
 
815
221
  ### 9. Execute tests
816
222
 
@@ -818,356 +224,90 @@ If playwright.config.ts exists → READ first, preserve ALL existing fields, add
818
224
  openspec-pw run <name> [options]
819
225
  ```
820
226
 
821
- **Available options** (from `src/index.ts:55-72`):
822
-
823
- | Option | Description |
824
- | --- | --- |
825
- | `-p, --project <name>` | Playwright project to run (e.g., user, admin) |
826
- | `-t, --timeout <seconds>` | Test timeout in seconds (default: 300) |
827
- | `--json` | Output results as JSON |
828
- | `-g, --grep <pattern>` | Run only tests matching pattern |
829
- | `--smoke` | Run only smoke tests (equivalent to `--grep @smoke`) |
830
- | `-w, --workers <n>` | Number of parallel workers |
831
- | `--app-bugs <n>` | Number of app bugs (skipped tests) |
832
- | `--healed <n>` | Number of test bugs healed by Healer |
833
- | `--raft <n>` | Number of RAFTs detected |
834
- | `--escalated <n>` | Number of human escalations |
835
- | `--headed` | Show browser during test run (default: headless) |
836
- | `--update-snapshots` | Update screenshot baselines before running tests |
837
-
838
- The CLI handles: server lifecycle, port mismatch, report generation.
839
-
840
- If tests fail → use Playwright MCP tools to inspect UI, fix selectors, re-run.
841
-
842
- **Browser visibility**: The Healer uses browser MCP tools (snapshot, screenshot, console messages) to inspect failures — no need for `--headed`. If you want to **watch the browser yourself** during debugging, add `--headed`: `openspec-pw run <name> --headed`.
843
-
844
- **Healer MCP tools** (in order of use):
845
-
846
- | Tool | Purpose |
847
- | -------------------------- | ----------------------------------------------- |
848
- | `browser_navigate` | Go to the failing test's page |
849
- | `browser_snapshot` | Get page structure to find equivalent selectors |
850
- | `browser_console_messages` | Diagnose JS errors that may cause failures |
851
- | `browser_network_requests` | Diagnose backend/API failures (4xx/5xx) |
852
- | `browser_take_screenshot` | Visually compare before/after fixes |
853
- | `browser_run_code` | Execute custom fix logic (optional) |
227
+ Flags: `-p --project`, `-t --timeout`, `--json`, `-g --grep`, `--smoke`, `-w --workers`, `--headed`, `--update-snapshots`.
854
228
 
855
- **Before Phase 1 check accumulated knowledge:**
229
+ When tests fail **Healer** (3 phases):
856
230
 
857
- Read `tests/playwright/app-knowledge.md` → **Selector Fixes** table. If the failing test's selector or route matches a known fix, use it directly (skip Phase 2. If partial match use as the top candidate in Phase 2-5a). If file does not exist → skip.
858
-
859
- **Healer — Phase 1: Triage**
860
-
861
- When a test fails, classify before attempting repair.
862
-
863
- **Batch Failure Detection — run this FIRST when multiple tests fail:**
864
-
865
- ```
866
- Collect ALL failing test names + their failure reasons.
867
- Group by: same route + same action + same Playwright error type (e.g., both "element not found" with the **same** root cause confirmed by browser_console_messages showing no JS error and browser_network_requests showing no 4xx/5xx). If root cause differs (e.g., element A missing vs element B in different section), treat as **separate** groups.
868
- If ≥2 tests fall into the same group:
869
- → Pause individual healing
870
- → Navigate to that route + perform the action manually
871
- → Check browser_console_messages + browser_network_requests
872
- → If console error or 4xx/5xx present:
873
- → This is an App Bug (backend/API change), NOT Test Bugs
874
- → Classify all tests in this group as App Bug
875
- → Skip all → record 1 App Bug in registry (not N bugs)
876
- → Skip the rest of individual Triage for this group
877
- → If timeout errors:
878
- → Retry one isolated: `npx playwright test --grep "<first-test-name>"` [--project <role>]
879
- → If isolated passes → **RAFT** (shared state coupling) → skip all → note RAFT in report
880
- → If isolated still fails → individual **Flaky** (each test independently flaky) → proceed with individual Triage for this group
881
- → If no console/network error and no timeout, but all still fail:
882
- → Likely a shared state issue → **RAFT** → skip all → note RAFT in report
883
- Proceed with individual Triage only for tests NOT in a batch failure group.
884
- ```
885
-
886
- **After Batch Detection, individual Triage:**
231
+ **Phase 1 Triage**: Classify each failure before repairing.
887
232
 
888
233
  | Failure Type | Signal | Classification | Action |
889
- | --- | --- | --- | --- |
890
- | **Network/Backend** | `net::ERR`, 4xx/5xx in console/network | **App Bug** | `test.skip()` + record in `app-bug-registry.md` |
891
- | **JS Runtime Error** | Console error (non-network) | **App Bug** | `test.skip()` + record in `app-bug-registry.md` |
892
- | **Auth Expired** | Redirected to login mid-test | **Flaky** | Re-run auth.setup → re-run |
893
- | **Redirect Loop** | `ERR_TOO_MANY_REDIRECTS`, URL keeps changing on each snapshot | **App Bug** | Check auth first (see Step 4.2 loop detection). If auth is the cause → re-run auth. Otherwise → `test.skip()` + App Bug Registry |
894
- | **Page Refresh Loop** | Excessive console errors (>10) after navigation, page unstable | **App Bug** | `test.skip()` + record in App Bug Registry |
895
- | **Selector Not Found** | Element not found | **Test Bug** | Phase 2 Healer |
896
- | **Assertion Mismatch** | Wrong content/value | **Ambiguous** | Phase 2 Healer |
897
- | **Timeout** | waitFor/evaluate timeout | **Flaky** | Retry isolated: `npx playwright test --grep "<test-name>"` (1×, not counted in heal attempts). If it passes isolated but fails in suite → **RAFT**. If it consistently times out → check framework: React 19 / Next.js App Router: add `page.waitForLoadState('networkidle')`. Vue/Angular/React 18 / Plain JS / jQuery: use `waitForSelector(targetElement)` instead of timeout tuning. |
898
- | **Same test fails in suite, passes isolated** | — | **RAFT** | `test.skip()` in suite, note RAFT in report |
899
-
900
- - **App Bug** → skip immediately (no healing needed) → record in App Bug Registry
901
- - **Flaky** → retry once isolated
902
- - **Test Bug / Ambiguous** → Phase 2
903
-
904
- #### App Bug Registry
905
-
906
- For every App Bug classified in Phase 1, record it in `openspec/reports/app-bug-registry.md` (create if missing):
907
-
908
- ```markdown
909
- # App Bug Registry
910
-
911
- <!-- Auto-generated. Do not edit manually. -->
912
-
913
- ## Active App Bugs
914
-
915
- | # | Test | Route | Signal | First Detected | Status |
916
- |---|------|-------|--------|---------------|--------|
917
- | 1 | test-name | /route | net::ERR_CONNECTION_REFUSED | 2026-04-09 | open |
918
- ```
919
-
920
- **Update rules**:
921
- - **New App Bug**: Append new row, increment `#`
922
- - **App Bug re-run and now passes**: Keep row, change status to `resolved` + add `Resolved` column with date
923
- - **Keep all rows** (never delete) — the resolved count is the signal that bugs are being fixed
924
-
925
- > **Why this matters**: `test.skip()` hides App Bugs from the pass/fail count. Without an explicit registry, "all tests passed" is a false positive when App Bugs exist. The registry makes the invisible visible.
926
-
927
- > **Global attempt guard**: Each test has an independent heal counter (max 3 per test). If the same test enters Phase 2 more than once and reaches the cap each time → treat as "consecutive escalation without progress" → Phase 3 immediately.
928
-
929
- > **Type ≠ Blame**: "Test Bug" means the assertion or selector is wrong — it does NOT mean "blame the test author." The test was generated from the spec. Root cause may be spec ambiguity, spec→test generation error, or app→spec deviation. Only a human can determine blame.
930
-
931
- **Healer — Phase 2: Repair**
932
-
933
- After Triage classifies failure as "Test Bug" or "Ambiguous":
934
-
935
- **Phase 2-0 — Batch diagnosis** (no Playwright, fast):
936
- For ALL Phase 2 failures in this run, do this simultaneously:
937
- 1. Read the failing test spec file(s) to understand what each test verifies
938
- 2. Read `app-knowledge.md` for any previous fixes or selector patterns
939
- 3. Read `app-knowledge.md` → **Common Selector Patterns** for project conventions
940
- 4. For each test, output:
941
- ```
942
- TEST: <test-name>
943
- ROUTE: <route>
944
- ASSERTION: "<what the test expects>"
945
- EXPECTED_BEHAVIOR: <from spec, one line>
946
- KNOWN_FIX: <yes/no — from app-knowledge.md Selector Fixes table>
947
- ```
948
- 5. Classify each test:
949
- - `ready-to-fix` — known fix exists, or selector pattern is clear
950
- - `needs-assertion-fix` — assertion itself needs changing (typo, wrong expected value)
951
- - `needs-phase3` — ambiguous, no clear fix path
952
- - `needs-more-diagnosis` — need to see actual page to determine
953
-
954
- **Skip remaining steps for `needs-phase3` tests** — go directly to Phase 3.
955
-
956
- **For `needs-assertion-fix` tests** — go to Phase 2-1 (navigate + snapshot → assertion fix).
957
-
958
- **For `ready-to-fix` tests** — go to Phase 2-5 (selector repair).
959
-
960
- **For `needs-more-diagnosis` tests** — go to Phase 2-1 (navigate + snapshot) for each individually.
961
-
962
- **Phase 2-1** (for `needs-assertion-fix` or `needs-more-diagnosis`): Navigate to the failing page → `browser_snapshot` → **EXPLICIT COMPARISON**:
963
- ```
964
- ASSERTION: "<what the test expects>"
965
- ACTUAL: "<what the snapshot shows>"
966
- MATCH: <yes/no>
967
- ```
968
- - **If MATCH=yes** → apply the assertion fix → go to **Phase 2-6**
969
- - **If MATCH=no**:
970
- - Safe to fix without Phase 3 → apply the fix → go to **Phase 2-6**
971
- - "Never fix without Phase 3" conditions met → **Phase 3** immediately
972
-
973
- **Assertion modification guard — never skip Phase 3 unless ALL conditions are met:**
974
- - The test has **never passed with the current assertion** (newly generated test, or this is the first time this assertion fails)
975
- - The ACTUAL value is **verifiably** from a different spec section (e.g., this test was in the wrong describe block)
976
- - You can point to the **specific line in the spec** that defines the ACTUAL behavior
234
+ | ------------ | ------ | -------------- | ------ |
235
+ | Network/Backend | `net::ERR`, 4xx/5xx | **App Bug** | `test.skip()` + App Bug Registry |
236
+ | JS Runtime Error | Console error | **App Bug** | `test.skip()` + App Bug Registry |
237
+ | Auth Expired | Redirected to /login | **Flaky** | Re-run auth.setup |
238
+ | Selector Not Found | Element not found | **Test Bug** | Phase 2 |
239
+ | Assertion Mismatch | Wrong content | **Ambiguous** | Phase 2 |
240
+ | Timeout | waitFor timeout | **Flaky** | Retry isolated |
241
+ | Same test: fails in suite, passes isolated | — | **RAFT** | `test.skip()` in suite, note in report |
977
242
 
978
- **If ANY condition is uncertainPhase 3 immediately.** Do NOT modify the assertion.
243
+ **Batch Detection**: When ≥2 tests fail with same route + error type, check console/network first. If backend error all App Bug (1 entry). If timeout check RAFT. If all same root cause → bulk classify.
979
244
 
980
- **Safe to fix without Phase 3:**
981
- - Typo in assertion (e.g., "Subm it" vs "Submit" in the expected text)
982
- - Selector was correct but the element was moved to a different location (same text, different selector)
983
- - Explicit spec drift confirmed by reading the spec (e.g., spec says "button says Submit" but test says "button says Submit Form")
245
+ **App Bug Registry**: `openspec/reports/app-bug-registry.md`. For each App Bug, append row (#, Test, Route, Signal, Date, Status). Never delete rows — resolved bugs change status to `resolved`.
984
246
 
985
- **Never fix without Phase 3:**
986
- - App behavior changed after an action (e.g., "after clicking submit, balance should decrease" → ACTUAL shows no change → **Phase 3**, could be optimistic update bug, backend failure, or spec mismatch)
987
- - Data values differ (e.g., expected "¥1000" but got "¥999" → **Phase 3**, could be rounding, discount, or calculation bug)
988
- - Missing elements after interaction (e.g., "after creating order, success message should appear" → no message → **Phase 3**)
247
+ **Phase 2 Repair** (for Test Bug / Ambiguous):
989
248
 
990
- **Phase 2-5** (for `ready-to-fix` with selector issue): Generate candidate list from snapshot:
249
+ 2-0. **Batch diagnosis**: Read failing test specs + app-knowledge.md fixes. Output per test: TEST, ROUTE, ASSERTION, EXPECTED_BEHAVIOR, KNOWN_FIX. Classify: `ready-to-fix` / `needs-assertion-fix` / `needs-phase3` / `needs-more-diagnosis`.
991
250
 
992
- **Phase 2-5a. Extract candidates** identify the target element from the failing test's assertion.
251
+ 2-1. **Navigate + snapshot**: ASSERTION vs ACTUAL comparison. Safe to fix without Phase 3: typo, moved element, confirmed spec drift. **Never fix without Phase 3**: behavior changed after action, data values differ, missing elements after interaction.
993
252
 
994
- If `KNOWN_FIX=yes` from Phase 2-0 diagnosis read the **New Selector** from `app-knowledge.md` **Selector Fixes** table (same Route + Old Selector row) apply directly → go to **Phase 2-6** (no need to re-generate candidates from snapshot).
253
+ 2-5. **Selector repair**: Pick highest-stability candidate: `getByRole` > `getByText/getByLabel` > `locator('#id')` > `locator('.class')` > `locator('nth-child')`.
995
254
 
996
- Otherwise list all selectors for the target element from the snapshot, **and** check `app-knowledge.md` **Common Selector Patterns** for project-specific conventions.
255
+ 2-6. **Fix + verify**: Apply fix `npx playwright test --grep "<test-name>"` → if passes, log to app-knowledge.md (Selector Fixes / Assertion Fixes). Max 3 heal attempts per test.
997
256
 
998
- ```
999
- Target: <element from failing assertion, e.g. "button with text 'Submit'">
1000
- Candidates (stable → fragile):
1001
- - getByRole(button, { name: 'Submit' }) ← Stable (semantic)
1002
- - getByText('Submit') ← Fair (unique text)
1003
- - getByLabel('Email') ← Fair (form fields)
1004
- - locator('#submit') ← Fair (id attribute)
1005
- - locator('.btn-primary') ← Fragile (style class) — upgrade if listed in Common Selector Patterns
1006
- - locator('button:nth-child(3)') ← Fragile (DOM order)
1007
- ```
1008
-
1009
- Stability: `getByRole` > `getByText`/`getByLabel` > `locator('#id')` > `locator('.class')` > `locator('nth-child')`. Upgrade stability if `app-knowledge.md` → **Common Selector Patterns** explicitly lists the selector as preferred for this project.
1010
-
1011
- **Phase 2-5b. Select top candidate** — pick the highest-stability candidate that matches the target. Output: `SELECTED: <selector> — reason: <why this one>`. Then → go to **Phase 2-6**.
1012
-
1013
- **Phase 2-6** — **Fix and verify incrementally (per test)**:
1014
- 1. For each test needing a fix, apply the fix to the `.spec.ts` file
1015
- 2. Run: `npx playwright test --grep "<test-name>"` to verify that specific test
1016
- 3. If passed → Phase 2-7 (log to app-knowledge.md), then move to next test
1017
- 4. If failed → analyze the new failure type:
1018
- - **Assertion-related** (wrong value/content) → return to **Phase 2-1** for that test
1019
- - **Selector-related** (element not found, wrong element) → return to **Phase 2-5** for that test (re-diagnose selector, not Phase 2-1)
1020
- - **Timeout-related** → treat as Flaky → retry isolated once: `npx playwright test --grep "<test-name>"` [--project <role>]
1021
- - If isolated passes → **Phase 2-7** (healed), then move to next test
1022
- - If isolated still fails → apply framework fix (React 19 / Next.js: `page.waitForLoadState('networkidle')`; Vue/Angular/React 18 / Plain JS: `waitForSelector(targetElement)`) → retry → if passes → **Phase 2-7** (healed) → move to next test; if still fails → `test.skip()` in `.spec.ts` + note in report → move to next test (not counted in heal attempts)
1023
- The heal counter does NOT reset for any path.
1024
- 5. Repeat until all Phase 2 tests are healed.
1025
-
1026
- > **Why --grep instead of --only-changed**: Healer fixing is incremental — you fix one test, verify it, adjust if needed, then move to the next. `--only-changed` is **file-level**: it runs ALL tests in `.spec.ts` files that have uncommitted source changes, not individual tests. This wastes time in an iterative fix cycle (re-running already-passed tests in the same file). Use `--grep` for targeted verification. Reserve `--only-changed` for the pre-commit guard.
1027
-
1028
- > **Pre-commit guard — `--only-changed`** (after all Phase 2 tests are healed): `npx playwright test --only-changed` runs all tests in changed `.spec.ts` files (comparing against HEAD by default, or `--only-changed=main` against a branch). Playwright also analyzes source file dependencies — if `src/components/Button.ts` changed, any `.spec.ts` that imports it will run. CI still runs the full suite as complete regression.
1029
-
1030
- **Phase 2-7** — If healed → append to `app-knowledge.md` (auto-de-duplicate):
1031
-
1032
- **Selector Fixes** — Before appending, read the existing **Selector Fixes** table. **Skip if** the same Route + Old Selector key already has a row (same route + same old selector = duplicate). If the Route + Old Selector exists but with a different New Selector → this is a new symptom of the same root cause → add a new row (different date, updated new selector).
1033
-
1034
- **Assertion Fixes** — Before appending, read the existing **Assertion Fixes** table. **Skip if** the same Test + Old Assertion key already has a row. If the same test had a previous fix with a different old → new → add a new row (the assertion has drifted further).
1035
-
1036
- If a new row is added, update the `Last updated` timestamp in the file header.
1037
-
1038
- This log feeds the **Auto-Heal Log** in the Phase 10 report.
1039
-
1040
- **Element Missing handling (when browser_snapshot shows element not found):**
1041
-
1042
- | Situation | Check | Action |
1043
- | --- | --- | --- |
1044
- | JS error in console after action | `browser_console_messages` | **App Bug** → Phase 1 → App Bug classification |
1045
- | Auth redirected mid-action | URL changed to `/login` | **Flaky** → re-run with fresh auth |
1046
- | SPA route didn't update | URL is correct but element missing | Wait for SPA hydration → `page.waitForLoadState('networkidle')` or `waitForSelector(target)` |
1047
- | Element genuinely missing | None of the above | **Test Bug** → find alternative selector or **Phase 3** if no equivalent exists |
1048
-
1049
- **Healer — Phase 3: Escalate**
1050
-
1051
- When Phase 2 tried ≥3 heals without success, OR ASSERTION vs ACTUAL comparison is ambiguous:
1052
-
1053
- **STOP** and output:
257
+ 2-7. **Log**: Append healed selector/assertion to `app-knowledge.md`. Auto-de-duplicate by Route + Old Selector (selectors) or Test + Old Assertion (assertions).
1054
258
 
259
+ **Phase 3 — Escalate** (after ≥3 heals or ambiguous comparison): STOP and output:
1055
260
  ```
1056
261
  E2E Test Failed — Human Decision Required
1057
-
1058
- Test: <test-name>
1059
- Failure: <type>
1060
- Assertion: "<what test expects>"
1061
- Actual: "<what app shows>"
1062
-
1063
- This failure could be:
1064
- 1. App does not match the spec → **app bug**
1065
- 2. Test was generated from ambiguous/incorrect spec → **spec issue**
1066
- 3. Spec itself is outdated (app was updated) → **spec drift**
1067
-
1068
- Please decide:
1069
- (a) Fix the app to match the spec
1070
- (b) Update the spec to match the app
1071
- (c) Update the test assertion
1072
- (d) Skip this test with test.skip() until resolved
262
+ Test: <name> | Failure: <type>
263
+ Assertion: "<expected>" | Actual: "<actual>"
264
+ Options: (a) Fix app, (b) Update spec, (c) Update assertion, (d) Skip
1073
265
  ```
1074
266
 
1075
- Wait for user input before proceeding.
267
+ Wait for user input. Track escalation attempts — if 3 consecutive Phase 3 with no progress, STOP and flag.
1076
268
 
1077
- **Decision tree follow the path based on user's choice:**
269
+ **Post-heal**: After all Phase 2 tests healed, run `npx playwright test --only-changed` as pre-commit guard.
1078
270
 
1079
- | Choice | What to do | After fix, do this |
1080
- |--------|-----------|-------------------|
1081
- | **(a)** Fix the app to match the spec | Fix the app code | Re-run: `openspec-pw run <change-name>` to verify fix |
1082
- | **(b)** Update the spec to match the app | Edit the spec file | Then update the test assertion (→ option c), or regenerate the affected part of the test |
1083
- | **(c)** Update the test assertion | Fix the assertion in `tests/playwright/changes/<name>/<name>.spec.ts` | Re-run: `openspec-pw run <change-name>` to verify |
1084
- | **(d)** Skip with `test.skip()` | Add `test.skip()` to the test | Note in `app-knowledge.md` → `Assertion Fixes` with reason "human escalation — skipped pending resolution" |
271
+ ### 10. Report results
1085
272
 
1086
- **Stuck in escalation loop** (see also: "Global attempt guard" above — tracked **per test**, independent heal counter): If 3 consecutive Phase 3 escalations for the same test result in no progress, STOP and ask: "This test has been escalated 3 times without resolution. Are you sure the root cause is still the same, or has something changed?"
273
+ Compile from `playwright-e2e-<name>-<timestamp>.md` and Phase 1–3 output:
274
+ - Summary table (App Bugs / Test Bugs healed / Flaky-RAFT / Escalations)
275
+ - App Bug Summary (with accumulation warning if ≥3 active)
276
+ - Failure Classification table
277
+ - Auto-heal log
278
+ - RAFT Summary
279
+ - Human Escalations
280
+ - Recommendations
1087
281
 
1088
- After the issue is resolved, re-run tests:
1089
- ```
1090
- openspec-pw run <change-name>
1091
- ```
1092
- `/opsx:e2e <change-name>` re-runs the full 10-step workflow — unnecessary after Phase 3. The test file and auth context are already correct. Use `openspec-pw run` to verify fixes directly.
1093
-
1094
- ### 10. False Pass Detection + RAFT Detection + App Bug Accumulation + Report results
1095
-
1096
- Run after test suite completes (even if all pass).
1097
-
1098
- **False Pass patterns** (test passed but shouldn't have):
1099
-
1100
- - **Conditional visibility**: `if (locator.isVisible().catch(() => false))` — if test passes, locator may not exist
1101
- - **Too fast**: < 200ms for a complex flow is suspicious
1102
- - **No fresh auth context**: Protected routes without `browser.newContext()`
1103
-
1104
- **RAFT detection** (Resource-Affected Flaky Test):
1105
-
1106
- - If you already ran the suite and a test failed: re-run that test in isolation with `npx playwright test --grep "<test-name>"` [--project <role>] → if it passes in isolation but fails in suite → **RAFT**
1107
- - This is **NOT** a test bug or app bug. Mark as RAFT, add `test.skip()` in suite, note in report
1108
- - RAFTs are infrastructure coupling issues (CPU/memory/I/O contention), not fixable by changing test or app
282
+ **Conditional "All Pass"**:
283
+ - ✅ **"All tests passed"** — 0 active App Bugs, 0 skipped
284
+ - ⚠️ **"All tests passed (N skipped)"** — skipped exist, no active App Bugs
285
+ - ⚠️ **"All tests passed (N skipped, M App Bugs unresolved)"** — active App Bugs exist
1109
286
 
1110
- **App Bug Accumulation Detection** (most critical):
1111
-
1112
- 1. Read `openspec/reports/app-bug-registry.md`
1113
- 2. Count `open` status rows
1114
- 3. If ≥ 3 active App Bugs → add "⚠️ App Bug accumulation: N bugs unresolved" to report summary. **Do NOT suppress or hide this warning.** The test suite may report "all passed" (skipped ≠ failed), but N broken features is not a passing system.
1115
- 4. Check for **App Bugs that became "resolved"** (passing on re-run) → this is the signal that bugs were fixed. Update `app-bug-registry.md` accordingly, and remove `test.skip()` from those tests so they re-enter the regression suite.
1116
-
1117
- **Report results** — compile from these sources (the auto-generated `playwright-e2e-<name>-<timestamp>.md` provides the Summary table; populate the sections below manually from Phase 1–3 output):
1118
-
1119
- - **Summary table** with failure type breakdown (App Bugs, Test Bugs/healed, Flaky-RAFT, Human Escalations) ← from `openspec/reports/playwright-e2e-<name>-<timestamp>.md`
1120
- - **App Bug Summary**: Table of all active App Bugs — test name, route, signal, first detected. If ≥ 3 active → display "⚠️ App Bug accumulation warning" prominently. ← from `openspec/reports/app-bug-registry.md`
1121
- - **Conditional "All Pass" conclusion**:
1122
- - ✅ **"All tests passed"** — only if 0 active App Bugs and 0 skipped tests
1123
- - ⚠️ **"All tests passed (N skipped)"** — if skipped tests exist but no active App Bugs
1124
- - ⚠️ **"All tests passed (N skipped, M App Bugs unresolved)"** — if active App Bugs exist
1125
- - **Failure Classification table**: (test, type, action, healed?) ← compile from Phase 1 Triage + Phase 2-0 classification output
1126
- - **Auto-heal log**: (assertion vs actual comparison, fix applied, result) ← compile from Phase 2-1/2-5 console output + `app-knowledge.md` Selector Fixes table
1127
- - **RAFT Summary** (if any): ← compile from RAFT detection notes during Phase 1
1128
- - **Human Escalations** (if any, with user decision): ← from Phase 3 decision output
1129
- - **Recommendations** with `file:line` references
1130
-
1131
- **Update tasks.md**:
1132
- - If 0 active App Bugs → find E2E-related items, append `✅ Verified via Playwright E2E (<timestamp>)`.
1133
- - If active App Bugs exist → **do not mark as verified**. Append instead: `⚠️ App Bug blocked: <bug summary> (<timestamp>)`.
287
+ **Update tasks.md**: If 0 active App Bugs → append `✅ Verified via Playwright E2E (<timestamp>)`. If App Bugs exist → append `⚠️ App Bug blocked: <summary> (<timestamp>)` (do not mark as verified).
1134
288
 
1135
289
  ## Graceful Degradation
1136
290
 
1137
- | Scenario | Classification | Action | Workflow Status |
1138
- | ------- | -------------- | ------ | --------------- |
1139
- | No specs / app-exploration.md missing (change mode) | Blocker | **STOP** — check path: `openspec/changes/<name>/specs/playwright/app-exploration.md` (change) or `<root>/app-exploration.md` (all mode) | Stops entirely |
1140
- | JS errors or HTTP 5xx during exploration | Blocker | **STOP** — user fixes app → re-run `/opsx:e2e <name>` from Step 4 | Stops entirely |
1141
- | Redirect loop / page refresh loop during exploration | App Bug | Skip route → record in App Bug Registry → re-run after fix | Continues (other routes) |
1142
- | File already exists (app-exploration, test-plan, app-all.spec.ts, Page Objects) | Idempotency | Read and use — never regenerate | Continues |
1143
- | Test fails (network/backend) | App Bug | `test.skip()` + record in `app-bug-registry.md` | Continues (workflow level — no STOP) |
1144
- | Test fails (selector/assertion) | Test Bug/Ambiguous | Healer Phase 1→2 (≤3 attempts) | Continues |
1145
- | RAFT detected (suite fail, isolated pass) | Flaky | `test.skip()` in suite, note RAFT in report | Continues |
1146
- | Phase 3 escalation | Human needed | **STOP** + present 4 options → wait for user decision | Stops entirely |
1147
- | False pass detected | Coverage Gap | Add "⚠️ Coverage Gap" to report | Continues |
1148
- | App Bug skip accumulation (≥3 active App Bugs) | Warning | Add "⚠️ App Bug accumulation: N bugs unresolved" to report. Do not suppress. | Continues |
1149
-
1150
- **"STOP"** = workflow halts and waits for user input (true stop).
1151
- **"Continues"** = workflow proceeds to next step (test-level skip, no user wait).
291
+ | Scenario | Classification | Action |
292
+ | -------- | -------------- | ------ |
293
+ | No specs / app-exploration missing (change mode) | Blocker | **STOP** |
294
+ | JS errors or HTTP 5xx during exploration | Blocker | **STOP** |
295
+ | Redirect/refresh loop during exploration | App Bug | Skip route → record in registry |
296
+ | File already exists | Idempotency | Read and use — never regenerate |
297
+ | Test fails (network/backend) | App Bug | `test.skip()` + registry |
298
+ | Test fails (selector/assertion) | Test Bug | Healer Phase 1→2 (≤3 attempts) |
299
+ | RAFT detected | Flaky | `test.skip()` in suite |
300
+ | Phase 3 escalation | Human needed | **STOP** wait for user |
301
+ | ≥3 active App Bugs | Warning | Add accumulation warning to report |
1152
302
 
1153
303
  ## Guardrails
1154
304
 
1155
- **Decision table:**
1156
-
1157
305
  | Rule | Why |
1158
- | --- | --- |
306
+ | ---- | --- |
1159
307
  | Read specs as source of truth | Generated tests must match requirements |
1160
308
  | Step 4 before Step 6 | Real DOM data → accurate selectors |
1161
309
  | Never contradict specs | E2E validates implementation, not design |
1162
- | Cap heal at 3 attempts | Prevents infinite loops |
310
+ | Cap heal at 3 attempts per test | Prevents infinite loops |
1163
311
  | Write runnable code, not TODOs | Placeholders fail CI |
1164
312
 
1165
- **Files you can write to:**
1166
- `tests/playwright/`, `openspec/changes/<name>/specs/playwright/`, `openspec/reports/`, `playwright.config.ts`, `auth.setup.ts`
1167
-
1168
- > `tests/playwright/` — spec files, Page Objects, auth, credentials, app-knowledge.md
1169
- > `openspec/changes/<name>/specs/playwright/` — app-exploration.md, test-plan.md (change mode)
1170
- > `<root>/` — app-exploration.md (change mode: INPUT for `openspec-pw explore`; all mode: detailed output)
1171
- > `openspec/reports/` — test reports, app-bug-registry.md
1172
-
1173
- **Never write to:** any other directory
313
+ **Write scope**: `tests/playwright/` (specs, Page Objects, auth, credentials, app-knowledge.md), `openspec/changes/<name>/specs/playwright/` (exploration, test plan), `<root>/app-exploration.md` (all mode), `openspec/reports/` (reports, bug registry), `playwright.config.ts`, `auth.setup.ts`. **Never write to any other directory.**