openspec-playwright 0.1.73 → 0.1.74

package/.claude/skills/openspec-e2e/SKILL.md

@@ -16,7 +16,9 @@ metadata:
 
 ## Output
 
-- **Test file**: `tests/playwright/<name>.spec.ts` (e.g. `app-all.spec.ts` for "all")
+- **Test file**: `tests/playwright/<name>.spec.ts`
+- **Page Objects** (all mode): `tests/playwright/pages/<Route>Page.ts`
+- **Auth setup**: `tests/playwright/auth.setup.ts` (if auth required)
 - **Auth setup**: `tests/playwright/auth.setup.ts` (if auth required)
 - **Report**: `openspec/reports/playwright-e2e-<name>-<timestamp>.md`
 - **Test plan**: `openspec/changes/<name>/specs/playwright/test-plan.md` (change mode only)
@@ -25,14 +27,14 @@ metadata:
 
 Two modes, same pipeline:
 
-| Mode   | Command            | Route source             | Output            |
-| ------ | ------------------ | ------------------------ | ----------------- |
-| Change | `/opsx:e2e <name>` | OpenSpec specs           | `<name>.spec.ts`  |
-| All    | `/opsx:e2e all`    | sitemap + homepage crawl | `app-all.spec.ts` |
+| Mode   | Command            | Route source             | Output                         |
+| ------ | ------------------ | ------------------------ | ------------------------------- |
+| Change | `/opsx:e2e <name>` | OpenSpec specs           | `<name>.spec.ts`                |
+| All    | `/opsx:e2e all`    | sitemap + homepage crawl | `pages/*.ts` (Page Objects)     |
 
 Both modes update `app-knowledge.md` and `app-exploration.md`. All `.spec.ts` files run together as regression suite.
 
-> **Role mapping**: Planner (Step 4–5) → test-plan.md; Generator (Step 6) → `.spec.ts` with verified selectors; Healer (Step 9) → repairs failures via MCP.
+> **Role mapping**: Planner (Step 4–5) → test-plan.md; Generator (Step 6) → `.spec.ts` + Page Objects; Healer (Step 9) → repairs failures via MCP.
 
 ## Testing principles
 
@@ -71,7 +73,8 @@ Can this be tested through the UI?
 
 **"all" mode** (`/opsx:e2e all` — no OpenSpec needed):
 
-- Announce: "Mode: full app exploration"
+- Announce: "Mode: full app exploration + Page Object discovery"
+- **Goal**: Discover new routes, extract selectors, and build `pages/*.ts` Page Objects — accumulated asset for future Change tests
 - Discover routes via:
   1. Navigate to `${BASE_URL}/sitemap.xml` (if exists)
   2. Navigate to `${BASE_URL}/` → extract all links from snapshot
@@ -138,6 +141,9 @@ browser_navigate → browser_console_messages → browser_snapshot → browser_t
 | JS error in console           | App runtime error                 | **STOP** — tell user: "Page has JS errors. Fix them, then re-run /opsx:e2e."                  |
 | HTTP 404                      | Route not in app (metadata issue) | Continue — mark `⚠️ route not found` in app-exploration.md                                    |
 | Auth required, no credentials | Missing auth setup                | Continue — skip protected routes, explore login page                                          |
+| Suspicious network request     | API returned 4xx/5xx             | Continue — mark `⚠️ API error: <endpoint> returned <code>` in app-exploration.md               |
+
+**Network monitoring**: After navigating, use `browser_network_requests` to check for failed API calls. Failed requests (status ≥ 400) on a route indicate an API/backend issue — record in `app-exploration.md` for reference.
 
 **For guest routes** (no auth):
 
@@ -190,11 +196,14 @@ From `browser_snapshot` + `browser_evaluate`, identify these special elements pe
 | ------- | --------------- | ---------------------------------------------- | ------------------- |
 | `<canvas>` | `role="img"`, `tagName="CANVAS"` | `canvas.getContext('2d'/'webgl')`, `width`, `height` | High |
 | `<iframe>` | `role="iframe"`, `src` attribute | `frameLocator` available | High |
+| CAPTCHA | `.g-recaptcha`, `.h-captcha`, `[data-sitekey]`, canvas+slider | recaptcha score via API (if configured) | High |
+| OTP / SMS | 6-digit input, countdown timer | Check if dev bypass exists | High |
 | Shadow DOM | `role="generic"` with no children | Check `shadowRoot` via evaluate | Medium |
 | Rich text editor | `[contenteditable]`, `role="textbox"` | `innerHTML`, `getContent()` | Medium |
 | Video / Audio | `role="application"` or name contains "video"/"audio" | `evaluate` checks both `<video>` and `<audio>` tags | Medium |
-| Date picker | specific `data-testid` or class patterns | Click triggers → evaluate value | Low (skip unless specs mention) |
+| File upload | `<input type="file">` | `accept` attribute, `multiple` flag | Medium |
 | Drag-and-drop | drag events in JS | Simulate DnD via coordinate clicks | Low |
+| Date picker | specific `data-testid` or class patterns | Click triggers → evaluate value | Low (skip unless specs mention) |
 | Infinite scroll | Dynamic row insertion | Count elements before/after scroll | Low |
 | WebSocket / SSE | No DOM signal | Check `browser_console_messages` for WS events | Low |
 
@@ -241,6 +250,29 @@ const isContentEditable = await browser_evaluate(() => {
   const el = document.querySelector('[contenteditable]');
   return !!el;
 });
+
+// CAPTCHA — detect type
+const captchaInfo = await browser_evaluate(() => {
+  const recaptcha = document.querySelector('.g-recaptcha, [data-sitekey]');
+  if (recaptcha) return { type: 'recaptcha', sitekey: recaptcha.getAttribute('data-sitekey') };
+  const hcaptcha = document.querySelector('.h-captcha');
+  if (hcaptcha) return { type: 'hcaptcha', sitekey: hcaptcha.getAttribute('data-sitekey') };
+  const turnstile = document.querySelector('[data-sitekey*="cloudflare"]');
+  if (turnstile) return { type: 'turnstile' };
+  const canvas = document.querySelector('canvas[class*="captcha"]');
+  if (canvas) return { type: 'canvas-captcha' };
+  const slider = document.querySelector('[class*="slider"], [class*="drag"]');
+  if (slider) return { type: 'slider-captcha' };
+  return null;
+});
+
+// OTP input — detect
+const otpInfo = await browser_evaluate(() => {
+  const inputs = document.querySelectorAll('input');
+  const otpInputs = Array.from(inputs).filter(i => i.maxLength === 1 && i.type === 'text' || i.type === 'tel');
+  if (otpInputs.length >= 4) return { type: 'otp-sms', digits: otpInputs.length };
+  return null;
+});
 ```
 
 Record findings in `app-exploration.md` → **Special Elements Detected** table.
@@ -297,7 +329,16 @@ Read `tests/playwright/app-knowledge.md` as context for cross-change patterns.
 
 ### 5. Generate test plan
 
-> **"all" mode: skip this step — go directly to Step 6.**
+> **"all" mode: skip this step.** No OpenSpec specs → no test-plan to generate. All mode skips test-plan verification — Page Objects are discovered incrementally from exploration, not from structured specs.
+
+**All mode — brief confirmation before Step 6:**
+```
+## All Mode: Page Object Discovery
+Discovered <N> routes (<M> guest, <K> protected)
+Special elements: <element summary>
+Ready to generate Page Objects for: <page-name>Page.ts, <page-name>Page.ts, ...
+Reply **yes** to proceed, or tell me to exclude routes or adjust strategies.
+```
 
 **Change mode — prerequisite**: If `openspec/changes/<name>/specs/playwright/app-exploration.md` does not exist → **STOP**. Run Step 4 (explore application) before generating tests. Without real DOM data from exploration, selectors are guesses and tests will be fragile.
 
@@ -311,13 +352,62 @@ Template: `templates/test-plan.md`
 
 **Idempotency**: If test-plan.md exists → read and use, do NOT regenerate.
 
+**⚠️ Human verification — STOP before generating code.**
+
+After creating (or reading existing) test-plan.md, **stop and display the test plan summary** for user confirmation:
+
+**Output format** — show the test plan in markdown directly in the conversation:
+
+````markdown
+## Test Plan Summary: `<change-name>`
+
+**Auth**: required / not required | Roles: ...
+
+### Test Cases
+- ✅ `<test-name>` — `<route>`, happy path
+- ✅ `<test-name>` — `<route>`, error path: `<error condition>`
+
+### Special Elements
+- ⚠️ **CAPTCHA** at `<route>` — strategy: `auth.setup bypass / skip / api-only`
+- ⚠️ **Canvas/WebGL** at `<route>` — strategy: screenshot + dimensions
+- ⚠️ **OTP** at `<route>` — strategy: test credentials / dev bypass
+- ⚠️ **Iframe** at `<route>` — strategy: frameLocator + assert inner content
+- ⚠️ **Video/Audio** at `<route>` — strategy: play() + assert !paused
+- ⚠️ **File Upload** at `<route>` — strategy: setInputFiles + assert upload
+- ⚠️ **Drag-and-Drop** at `<route>` — strategy: dragAndDrop or evaluate events
+- ⚠️ **WebSocket/SSE** at `<route>` — strategy: waitForResponse + waitForFunction
+
+### Not Covered
+- `<element or scenario not testable>`
+````
+
+Then ask: "Does this coverage match your intent? Reply **yes** to proceed, or tell me what to add/change."
+
+**Why this matters**: Step 5 is the last human-reviewable checkpoint before code generation. Once test code is written, fixes address *how* tests run, not *what* they verify. Reviewing the test plan takes seconds and catches logic errors that Healer cannot fix.
+
+**Confirmation criteria**:
+- All scenarios from OpenSpec specs are covered
+- Special elements (Canvas, Iframe, Video, Audio, CAPTCHA, OTP, File Upload, Drag-drop, WebSocket) have correct automation strategy
+- Auth states and roles are accurate
+- Nothing important is missing
+
+If the user requests changes → update test-plan.md → re-display summary → re-confirm → proceed.
+
 ### 6. Generate test file
 
-**"all" mode** → `tests/playwright/app-all.spec.ts` (smoke regression):
+**"all" mode**: Build and expand Page Objects for future Change tests.
+
+**Prerequisite**: If `app-exploration.md` does not exist → **STOP**. Run Step 4 first. All mode explores routes via browser MCP to build exploration data.
+
+For each discovered route:
 
-- For each discovered route: navigate → assert HTTP 200 → assert ready signal visible
-- No detailed assertions — just "this page loads without crashing"
-- This is a regression baseline — catches when existing pages break
+1. Read existing `pages/<Route>Page.ts` (if any — incremental, not overwrite)
+2. Navigate to route with correct auth state
+3. browser_snapshot to extract interactive elements (see 4.3 table)
+4. Write or update `pages/<Route>Page.ts` — extend with newly discovered elements
+5. Also write `tests/playwright/app-all.spec.ts` — smoke test (route loads without crash)
+
+**Output priority**: Page Objects (`pages/*.ts`) are the primary asset. Smoke test is secondary. Existing Page Objects are never overwritten — only extended.
 
 **Change mode** → `tests/playwright/<name>.spec.ts` (functional):
 
@@ -399,9 +489,48 @@ test('video can be played', async ({ page }) => {
   const isPlaying = await video.evaluate((v: HTMLVideoElement) => !v.paused);
   expect(isPlaying).toBe(true);
 });
+
+// Audio — playback state
+test('audio can be played', async ({ page }) => {
+  await page.goto(`${BASE_URL}/<route>`);
+  const audio = page.locator('audio');
+  await expect(audio).toBeVisible();
+  await audio.evaluate((a: HTMLAudioElement) => { a.play(); });
+  const isPlaying = await audio.evaluate((a: HTMLAudioElement) => !a.paused);
+  expect(isPlaying).toBe(true);
+});
 ```
 
-See `templates/test-plan.md` → **Special Element Test Cases** for full templates.
+See `templates/test-plan.md` → **Special Element Test Cases** for full templates including Canvas, Video, Audio, Iframe, and Rich Text Editor.
+
+**Test coverage — AI-opaque elements**: For CAPTCHA, OTP, slider CAPTCHA, file upload, and drag-drop — elements that Playwright cannot reliably automate:
+
+1. Mark the element in `app-exploration.md` → **Special Elements Detected** table with type and automation strategy
+2. Generate the test using the appropriate strategy from `templates/test-plan.md` → **AI-Opaque Elements** section:
+   - **CAPTCHA**: Bypass via `auth.setup.ts` storageState, or skip with `test.skip()`, or verify via API
+   - **OTP**: Use pre-verified test credentials (`E2E_OTP_CODE` env var), or development bypass flag
+   - **File upload**: Use `page.setInputFiles()` with fixture files
+   - **Drag-drop**: Use `page.dragAndDrop()` or `page.evaluate()` with custom event dispatching
+3. If the element is truly non-automatable, write `test.skip()` with a comment explaining why, and mark with `/handoff` for manual testing
+
+**Test coverage — performance**: Verify Core Web Vitals metrics. If the app specifies performance targets, generate a test:
+
+```typescript
+// Performance — Core Web Vitals
+test('page loads within performance budget', async ({ page }) => {
+  await page.goto(`${BASE_URL}/<route>`);
+  await expect(page.getByRole('heading')).toBeVisible();
+  const timings = await page.evaluate(() => {
+    const nav = performance.getEntriesByType('navigation')[0] as PerformanceNavigationTiming;
+    return {
+      ttfb: nav.responseStart - nav.requestStart,
+      lcp: nav.loadEventEnd - nav.requestStart,
+    };
+  });
+  expect(timings.ttfb).toBeLessThan(500);
+  expect(timings.lcp).toBeLessThan(2500);
+});
+```
 
 ```typescript
 // 🚫 Avoid for special elements:
@@ -599,13 +728,12 @@ If tests fail → use Playwright MCP tools to inspect UI, fix selectors, re-run.
 
 **Healer MCP tools** (in order of use):
 
-<!-- MCP_VERSION: 0.0.70 -->
-
 | Tool                       | Purpose                                         |
 | -------------------------- | ----------------------------------------------- |
 | `browser_navigate`         | Go to the failing test's page                   |
 | `browser_snapshot`         | Get page structure to find equivalent selectors |
 | `browser_console_messages` | Diagnose JS errors that may cause failures      |
+| `browser_network_requests` | Diagnose backend/API failures (4xx/5xx)          |
 | `browser_take_screenshot`  | Visually compare before/after fixes             |
 | `browser_run_code`         | Execute custom fix logic (optional)             |
 
@@ -616,7 +744,7 @@ If tests fail → use Playwright MCP tools to inspect UI, fix selectors, re-run.
 
 | Failure type                 | Signal                                  | Action                                                |
 | ---------------------------- | --------------------------------------- | ----------------------------------------------------- |
-| **Network/backend**          | `fetch failed`, `net::ERR`, 5xx         | `browser_console_messages` → `test.skip()`            |
+| **Network/backend**          | `fetch failed`, `net::ERR`, 4xx/5xx | `browser_network_requests` → `browser_console_messages` → `test.skip()` |
 | **Selector changed**         | Element not found                       | `browser_snapshot` → fix selector → re-run            |
 | **Assertion mismatch**       | Wrong content/value                     | `browser_snapshot` → compare → fix assertion → re-run |
 | **Timing issue**             | `waitFor`/`page.evaluate` timeout       | Switch to `request` API or add `waitFor` → re-run     |
@@ -656,7 +784,7 @@ Reference: `templates/report.md`
 | No specs / app-exploration.md missing (change mode) | **STOP** |
 | JS errors or HTTP 5xx during exploration | **STOP** |
 | Sitemap fails ("all" mode) | Continue with homepage links fallback |
-| File already exists (app-exploration, test-plan, app-all) | Read and use — never regenerate |
+| File already exists (app-exploration, test-plan, app-all.spec.ts, Page Objects) | Read and use — never regenerate |
 | Test fails (backend) | `test.skip()` + report |
 | Test fails (selector/assertion) | Healer: snapshot → fix → re-run (≤3) |
 | 3 heals failed | `test.skip()` if app bug; report if unclear |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "openspec-playwright",
-  "version": "0.1.73",
+  "version": "0.1.74",
   "description": "OpenSpec + Playwright E2E verification setup tool for Claude Code",
   "type": "module",
   "bin": {

package/templates/app-exploration.md

@@ -46,7 +46,9 @@ BASE_URL: <from env or seed.spec.ts>
 | | | | | |
 | | | | | |
 
-> **Special element type legend**: `canvas-2d` | `canvas-webgl` | `iframe` | `shadow-dom` | `contenteditable` | `video` | `audio` | `datepicker` | `drag-drop` | `infinite-scroll`
+> **Special element type legend**: `canvas-2d` | `canvas-webgl` | `iframe` | `shadow-dom` | `contenteditable` | `video` | `audio` | `datepicker` | `drag-drop` | `infinite-scroll` | `file-upload` | `captcha-image` | `captcha-slider` | `captcha-3d` | `recaptcha` | `hcaptcha` | `turnstile` | `otp-sms` | `otp-totp` | `websocket` | `sse`
+>
+> **AI-opaque strategy**: CAPTCHA/OTP/slider-CAPTCHA — see `templates/test-plan.md` → **AI-Opaque Elements**
 >
 > **Test strategy**: See `templates/test-plan.md` → **Special Element Test Cases**
 

package/templates/test-plan.md

@@ -98,4 +98,156 @@ Generated from: `openspec/changes/<change-name>/specs/`
 2. Call `audio.play()` via `page.evaluate()`
 3. Assert `!audio.paused`
 
+## AI-Opaque Elements: Out of Scope for Automation
+
+<!-- Elements that AI cannot reliably interact with. Mark in `app-exploration.md` → Special Elements table. -->
+
+### CAPTCHA — Human Verification Required
+
+- **Route**: `/<page>`
+- **Type**: `captcha-image | captcha-slider | captcha-3d | recaptcha | hcaptcha | turnstile`
+- **Element**: `<canvas id="...">`, `.g-recaptcha`, `.h-captcha`, `[data-sitekey]`, slider track
+
+**Automation strategy — choose one:**
+
+#### Strategy A: Auth-Setup Bypass (Recommended)
+Use pre-authenticated sessions from `auth.setup.ts` so CAPTCHA is bypassed before the test runs.
+
+```
+1. Store authenticated storageState in auth.setup.ts
+2. In test: use storageState to skip login
+3. Verify post-CAPTCHA state via API instead
+4. Assert: API response after CAPTCHA = expected state
+```
+
+**Example assertions:**
+- `API POST /submit-form` returns 200 + `success: true`
+- UI redirects to `/dashboard` after CAPTCHA pass
+- Database record created with correct values
+
+#### Strategy B: Skip with /handoff
+If CAPTCHA is the primary interaction under test, mark as non-automatable:
+
+```
+test.skip('<feature> requires CAPTCHA', async ({ page }) => {
+  // Human reviewer must manually pass CAPTCHA
+  // Then run: npx playwright test --grep "<feature>"
+});
+```
+
+#### Strategy C: API Verification Only
+Test the result of CAPTCHA-protected actions without automating the CAPTCHA:
+
+```
+1. Manually pass CAPTCHA once → capture resulting session/token
+2. Use that session in subsequent API calls
+3. Assert: API behavior matches CAPTCHA-passed state
+```
+
+**Detected CAPTCHA type** (fill in based on `app-exploration.md`):
+- Type: `<!-- image | slider | 3d-object | recaptcha | hcaptcha | turnstile -->`
+- Bypass method: `<!-- auth.setup | skip | api-only -->`
+- Verified by: `<!-- human-manual | api-response | db-state -->`
+
+---
+
+### OTP / SMS Verification — Test Account Bypass
+
+- **Route**: `/<page>`
+- **Type**: `otp-sms | otp-email | totp`
+- **Element**: 6-digit input, countdown timer, resend button
+
+**Automation strategy:**
+
+#### Strategy A: Test Credentials (Recommended)
+Use pre-verified test accounts with known OTP codes.
+
+```
+1. Set E2E_OTP_CODE=<valid-test-code> in credentials.yaml
+2. In test:
+   await page.getByRole('textbox').fill(process.env.E2E_OTP_CODE);
+   await page.getByRole('button', { name: 'Verify' }).click();
+```
+
+#### Strategy B: Development Bypass Flag
+If dev mode disables OTP, use that flag.
+
+```
+1. Set BASE_URL to dev environment with OTP disabled
+2. Proceed as normal authenticated user
+```
+
+#### Strategy C: API-Only Verification
+Bypass OTP UI entirely and test the protected endpoint directly.
+
+```
+1. Obtain valid token via API (skip OTP step)
+2. Use token in subsequent API calls
+3. Assert: protected endpoint returns expected data
+```
+
+---
+
+### File Upload — Complex Input
+
+- **Route**: `/<page>`
+- **Type**: `file-upload`
+- **Element**: `<input type="file">`, drag-and-drop zone
+
+**Test approach:**
+```
+1. Create test fixture file: `test fixtures/login-hero.png`
+2. Set `accept: '<mime-type>'` if specified
+3. Use `page.setInputFiles()` for reliable upload
+4. Assert: upload progress completes → success message or preview
+```
+
+**Assertions:**
+- Upload progress bar completes (if shown)
+- File preview renders correctly
+- API response contains uploaded file reference
+
+---
+
+### Drag-and-Drop — Custom Implementation
+
+- **Route**: `/<page>`
+- **Type**: `drag-drop`
+- **Element**: `[draggable]`, `.drop-zone`, sortable list items
+
+**Test approach:**
+```
+1. Identify drag handle and drop target via Playwright MCP snapshot
+2. Use `page.dragAndDrop()` or `locator.dragTo()`
+3. If custom implementation uses JS events, use `page.evaluate()` to dispatch events
+4. Assert: target state reflects the drag result
+```
+
+**Note**: If the drag-drop uses complex physics (e.g., Kanban board, calendar), prefer `page.evaluate()` with custom event dispatching over `dragAndDrop()`.
+
+---
+
+### WebSocket / Real-Time — Timing Sensitive
+
+- **Route**: `/<page>`
+- **Type**: `websocket | sse | polling`
+- **Element**: live data display, notification badge, live chart
+
+**Test approach:**
+```
+1. Establish WebSocket/SSE connection via page
+2. Wait for specific message/event: `await page.waitForResponse(/<ws-endpoint>/)`
+3. Assert: DOM updates after message received
+4. Use `page.waitForFunction()` to poll for expected state
+```
+
+**Assertions:**
+- Live data counter increments after server push
+- Notification badge shows correct count
+- Chart redraws with new data points
+
+**Note**: Increase test timeout for real-time tests. See `playwright.config.ts` → `timeout`.
+
+---
+
 > **Reference**: See `app-exploration.md` → **Special Elements Detected** table for per-route specifics.