qualiow-exploratory-testing 1.3.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +362 -205
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/knowledge/changelog.yml +30 -12
- package/data/knowledge/manifest.yml +333 -127
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +3 -3
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +9 -6
- package/data/templates/expected-behaviour.md +3 -2
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1452 -700
- package/dist/index.d.ts +589 -81
- package/dist/index.js +1612 -227
- package/docs/BACKEND-VERIFICATION.md +39 -9
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -18
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +26 -17
- package/skills/qa-verify-backend/phases/00-setup.md +57 -12
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +2 -2
- package/skills/qa-verify-backend/phases/03-live-verification.md +3 -3
- package/skills/qa-verify-backend/phases/03b-api-verification.md +22 -9
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +18 -2
- package/skills/qa-verify-backend/phases/05-reporting.md +38 -10
- package/skills/qa-verify-backend/references/api-probes.md +28 -1
- package/skills/qa-verify-backend/references/payload-verification.md +1 -1
- package/skills/qa-verify-backend/references/release-readiness.md +1 -1
- package/skills/qa-verify-backend/references/safety-rules.md +13 -8
- package/data/domains/_default.md +0 -181
- package/data/domains/ecommerce.md +0 -192
- package/data/domains/fintech.md +0 -200
- package/data/domains/identity.md +0 -88
- package/data/domains/marketing.md +0 -194
- package/data/domains/saas.md +0 -197
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# Phase 1: Authentication
|
|
1
|
+
# Phase 1: Authentication (2 min)
|
|
2
2
|
|
|
3
|
-
Skip if `auth.strategy: none`.
|
|
3
|
+
Skip if `auth.strategy: none`. Every command below carries the session id `-s=<sid>` (omitted here).
|
|
4
4
|
|
|
5
5
|
## Storage State (fastest)
|
|
6
6
|
|
|
@@ -9,30 +9,47 @@ playwright-cli state-load .auth/<target>.json
|
|
|
9
9
|
playwright-cli goto <base_url>
|
|
10
10
|
```
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
If the state file is older than 24 hours, expect it to have expired and fall through to
|
|
13
|
+
credentials when the login page appears.
|
|
13
14
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
15
|
+
## Credentials (adaptive: uses snapshot to find form fields)
|
|
16
|
+
|
|
17
|
+
Resolve the values named by `auth.credentials.username` / `auth.credentials.password` in the
|
|
18
|
+
order from `${CLAUDE_SKILL_DIR}/references/paths.md` (`qa/.env`, then `.env`). Read the value,
|
|
19
|
+
pass it to the command, and never echo it into the log or the report.
|
|
17
20
|
|
|
18
21
|
```bash
|
|
19
22
|
playwright-cli open <login_url>
|
|
20
23
|
playwright-cli snapshot
|
|
21
|
-
playwright-cli fill <email_ref> "
|
|
22
|
-
playwright-cli fill <pass_ref> "
|
|
24
|
+
playwright-cli fill <email_ref> "<username value>"
|
|
25
|
+
playwright-cli fill <pass_ref> "<password value>"
|
|
23
26
|
playwright-cli click <submit_ref>
|
|
24
27
|
playwright-cli state-save .auth/<target>.json
|
|
25
28
|
```
|
|
26
29
|
|
|
30
|
+
## Token (`auth.strategy: token`)
|
|
31
|
+
|
|
32
|
+
Inject the token named by `auth.token` where the app expects it, then reload:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
playwright-cli open <base_url>
|
|
36
|
+
playwright-cli cookie-set <cookie-name> "<token value>" --domain=<host> --httpOnly --secure # cookie-based
|
|
37
|
+
playwright-cli localstorage-set <key> "<token value>" # localStorage-based
|
|
38
|
+
playwright-cli reload
|
|
39
|
+
```
|
|
40
|
+
|
|
27
41
|
## Verify Login
|
|
28
42
|
|
|
29
|
-
After authentication,
|
|
43
|
+
After authentication, `playwright-cli snapshot` and confirm user indicators are present
|
|
44
|
+
(username display, avatar, dashboard access).
|
|
30
45
|
|
|
31
46
|
If login fails:
|
|
32
|
-
1. Check console
|
|
33
|
-
2.
|
|
34
|
-
3.
|
|
35
|
-
4.
|
|
36
|
-
|
|
37
|
-
|
|
47
|
+
1. Check the console: `playwright-cli console error`
|
|
48
|
+
2. Check the last requests: `playwright-cli requests --filter="login|auth|session"`
|
|
49
|
+
3. Verify the credentials are the ones the target names
|
|
50
|
+
4. Check for MFA. Never automate an MFA challenge; hand it to the user in a headed browser
|
|
51
|
+
(`playwright-cli open <login_url> --headed`) and continue once they confirm
|
|
52
|
+
5. Log the failure and attempt an alternative strategy if the target defines one
|
|
53
|
+
|
|
54
|
+
**Update progress:** set the auth phase complete (or failed) in `progress.json`. Append to `session-log.md`:
|
|
38
55
|
`[<timestamp>] [PHASE] Auth complete — strategy: <strategy>, result: <success/failed>, user: <redacted>`
|
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
# Phase 2: Business Context & Charter (
|
|
1
|
+
# Phase 2: Business Context & Charter (5 min)
|
|
2
2
|
|
|
3
3
|
**This is the most important phase. Skip it and everything else is shallow.**
|
|
4
4
|
|
|
5
5
|
## Step 1: Use the App as a Real User
|
|
6
6
|
|
|
7
|
-
Before any testing, spend 2
|
|
7
|
+
Before any testing, spend 2 minutes USING the app the way a customer would. Not testing -- USING.
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
10
|
playwright-cli open <url>
|
|
@@ -62,9 +62,12 @@ Also select 2 Test Tours:
|
|
|
62
62
|
|
|
63
63
|
## Step 5: Write Charter
|
|
64
64
|
|
|
65
|
-
Save to `output/sessions/<session-dir>/charter.md
|
|
65
|
+
Save to `output/sessions/<session-dir>/charter.md` (confidentiality header first, per `${CLAUDE_SKILL_DIR}/references/output-contract.md`):
|
|
66
66
|
|
|
67
67
|
```markdown
|
|
68
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
69
|
+
> and application details. Do not share outside your organization without review.
|
|
70
|
+
|
|
68
71
|
# Session Charter
|
|
69
72
|
|
|
70
73
|
## Context Source
|
|
@@ -101,12 +104,20 @@ Save to `output/sessions/<session-dir>/charter.md`:
|
|
|
101
104
|
- FEW HICCUPPS oracles: Focus on [Users, Claims, Product]
|
|
102
105
|
|
|
103
106
|
## Session Parameters
|
|
104
|
-
- Time box:
|
|
107
|
+
- Time box: 45 min
|
|
105
108
|
- Domain: [domain]
|
|
106
109
|
- Focus: [focus area or "full exploration"]
|
|
110
|
+
- Read-only: [true | false]
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Start evidence capture (both carry `-s=<sid>`):
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
playwright-cli tracing-start
|
|
117
|
+
playwright-cli video-start output/sessions/<session-dir>/videos/session.webm
|
|
107
118
|
```
|
|
108
119
|
|
|
109
|
-
|
|
120
|
+
Add `playwright-cli video-chapter "<phase or feature>"` at each phase boundary so the recording is navigable.
|
|
110
121
|
|
|
111
122
|
**Update progress:** Set charter phase complete in progress.json. Append to session-log.md:
|
|
112
123
|
`[<timestamp>] [PHASE] Charter complete — risk ranking defined, <N> journeys planned, heuristics: <selected>`
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# Phase 3: Discovery & Mapping (
|
|
1
|
+
# Phase 3: Discovery & Mapping (6 min)
|
|
2
2
|
|
|
3
|
-
**Goal:** Map the app structure and refine your risk ranking.
|
|
3
|
+
**Goal:** Map the app structure and refine your risk ranking. Every command carries `-s=<sid>` (omitted here).
|
|
4
4
|
|
|
5
5
|
## Site Mapping
|
|
6
6
|
|
|
@@ -8,56 +8,80 @@
|
|
|
8
8
|
playwright-cli goto <start_url>
|
|
9
9
|
playwright-cli snapshot
|
|
10
10
|
playwright-cli console error
|
|
11
|
-
playwright-cli
|
|
11
|
+
playwright-cli requests
|
|
12
12
|
```
|
|
13
13
|
|
|
14
14
|
For each page:
|
|
15
|
-
1. `playwright-cli snapshot`
|
|
16
|
-
2. Click through navigation to discover all sections
|
|
15
|
+
1. `playwright-cli snapshot` — note structure, forms, interactive elements. Use `snapshot --depth=3` on very large pages, then `snapshot <ref>` for the region you care about.
|
|
16
|
+
2. Click through navigation to discover all sections.
|
|
17
17
|
3. Log: `[DISCOVERED] <url> -- <description> -- Risk: <P0/P1/P2/P3>`
|
|
18
|
-
4. Check `playwright-cli console error` on each page
|
|
18
|
+
4. Check `playwright-cli console error` and `playwright-cli requests` on each page.
|
|
19
|
+
|
|
20
|
+
Useful helpers:
|
|
21
|
+
- `playwright-cli find "<text>"` (or `find --regex "<re>"`) — locate a label or message in the snapshot without reading the whole tree.
|
|
22
|
+
- `playwright-cli --raw snapshot > before.yml`, act, `playwright-cli --raw snapshot > after.yml`, then `diff before.yml after.yml` — see exactly what an action changed.
|
|
23
|
+
- `playwright-cli requests --filter="/api/"` — only the calls that matter; `playwright-cli request <n>` for headers and body of one of them.
|
|
19
24
|
|
|
20
25
|
## Scope Enforcement
|
|
21
26
|
|
|
22
|
-
Stay within the target domain. Do not follow links to external sites unless they are part of
|
|
27
|
+
Stay within the target domain. Do not follow links to external sites unless they are part of
|
|
28
|
+
the application's workflow (OAuth redirects, payment gateways).
|
|
23
29
|
|
|
24
30
|
## What's MISSING Check
|
|
25
31
|
|
|
26
|
-
**Critical during discovery
|
|
27
|
-
- Read the domain completeness checklist from
|
|
28
|
-
- Compare what the app HAS
|
|
29
|
-
- E-commerce missing payment method? Log it now.
|
|
32
|
+
**Critical during discovery — look for what's MISSING:**
|
|
33
|
+
- Read the domain completeness checklist from `<data>/domains/<domain>.yml` (`completeness_checklist`)
|
|
34
|
+
- Compare what the app HAS against what it SHOULD have
|
|
35
|
+
- E-commerce missing a payment method? Log it now.
|
|
30
36
|
- Banking app missing transaction history? Flag it immediately.
|
|
31
37
|
- No error messages on empty states? Note it.
|
|
32
38
|
|
|
33
39
|
## Console & Network Checks
|
|
34
40
|
|
|
35
41
|
On every page during discovery:
|
|
36
|
-
- `playwright-cli console error`
|
|
37
|
-
- `playwright-cli
|
|
38
|
-
- Watch for mixed
|
|
42
|
+
- `playwright-cli console error` — JavaScript errors
|
|
43
|
+
- `playwright-cli requests` — failed requests (4xx, 5xx); `requests --static` to include assets
|
|
44
|
+
- Watch for mixed-content warnings (HTTP resources on HTTPS pages)
|
|
45
|
+
|
|
46
|
+
### Cross-navigation observation via `run-code` (Playwright 1.59+)
|
|
39
47
|
|
|
40
|
-
|
|
48
|
+
The CLI's `console` and `requests` show only the tail since the last navigation. To reach
|
|
49
|
+
messages from *before* it, run Playwright's retrieval methods inside the session with
|
|
50
|
+
`run-code` (the callback receives the live `page`) and pass `filter: 'all'` — **without it
|
|
51
|
+
they default to `since-navigation` and return exactly the same window as the CLI**. The
|
|
52
|
+
buffers are capped: 200 console messages, 200 page errors, 100 requests.
|
|
41
53
|
|
|
42
|
-
|
|
54
|
+
```bash
|
|
55
|
+
# every console message still buffered, across navigations
|
|
56
|
+
playwright-cli run-code "async page => (await page.consoleMessages({ filter: 'all' })).map(m => m.type() + ': ' + m.text()).join('\n')"
|
|
43
57
|
|
|
44
|
-
|
|
45
|
-
-
|
|
46
|
-
|
|
58
|
+
# uncaught exceptions only — more reliable than filtering console output
|
|
59
|
+
playwright-cli run-code "async page => (await page.pageErrors({ filter: 'all' })).map(e => e.message).join('\n')"
|
|
60
|
+
|
|
61
|
+
# failures and error statuses among the last 100 requests (page.requests() takes no filter)
|
|
62
|
+
playwright-cli run-code "async page => { const out = []; for (const r of await page.requests()) { const res = await r.response(); const f = r.failure(); if (f) out.push('FAILED ' + f.errorText + ' ' + r.url()); else if (res && res.status() >= 400) out.push(res.status() + ' ' + r.url()); } return out.join('\n'); }"
|
|
63
|
+
```
|
|
47
64
|
|
|
48
|
-
These are
|
|
65
|
+
These are optional; the bash commands above remain the primary path.
|
|
49
66
|
|
|
50
|
-
### Service Workers (Playwright 1.
|
|
67
|
+
### Service Workers (Playwright 1.61+)
|
|
51
68
|
|
|
52
|
-
|
|
69
|
+
Service-worker requests are reported and routable at the **BrowserContext** level from 1.61.
|
|
70
|
+
`playwright-cli route` maps to `page.route()`, which per the API docs still does **not**
|
|
71
|
+
intercept service-worker traffic. So if `playwright-cli requests` shows calls you cannot
|
|
72
|
+
account for on the main document, assume a service worker is serving them: flag the area as
|
|
73
|
+
high-risk, and note in the report that mocking it needs service workers blocked
|
|
74
|
+
(`serviceWorkers: 'block'`), not `route`.
|
|
53
75
|
|
|
54
76
|
## After Discovery
|
|
55
77
|
|
|
56
|
-
1. Write findings to `phase-
|
|
78
|
+
1. Write findings to `phase-3-discovery.md` (confidentiality header first)
|
|
57
79
|
2. **Update your risk ranking** if discovery changed your understanding
|
|
58
|
-
3. Carry forward: site
|
|
80
|
+
3. Carry forward: site-map summary, risk ranking, list of what's missing
|
|
59
81
|
|
|
60
|
-
**Log your reasoning:** "I expected a payment
|
|
82
|
+
**Log your reasoning:** "I expected a payment-method selector in checkout because this is
|
|
83
|
+
e-commerce, but it's missing. I'll flag this as a medium bug."
|
|
61
84
|
|
|
62
|
-
**Update progress:**
|
|
85
|
+
**Update progress:** set the discovery phase complete in `progress.json` with the
|
|
86
|
+
`pages_explored` count. Append to `session-log.md`:
|
|
63
87
|
`[<timestamp>] [PHASE] Discovery complete — <N> pages mapped, <N> console errors, <N> missing checklist items, <N> bugs so far`
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Phase 4: End-to-End User Journeys (
|
|
1
|
+
# Phase 4: End-to-End User Journeys (10 min)
|
|
2
2
|
|
|
3
3
|
**Goal:** Test the critical user journeys you defined in the charter -- end to end, crossing multiple features.
|
|
4
4
|
|
|
@@ -43,7 +43,7 @@ Do things change? Are order IDs unique? Are timestamps correct?
|
|
|
43
43
|
|
|
44
44
|
## After Journeys
|
|
45
45
|
|
|
46
|
-
1. Write findings to `phase-
|
|
46
|
+
1. Write findings to `phase-4-journeys.md` (confidentiality header first)
|
|
47
47
|
2. Include data integrity results
|
|
48
48
|
3. Note any cross-feature inconsistencies
|
|
49
49
|
|
|
@@ -1,8 +1,9 @@
|
|
|
1
|
-
# Phase 5: Deep Feature Testing (
|
|
1
|
+
# Phase 5: Deep Feature Testing (10 min)
|
|
2
2
|
|
|
3
|
-
**Goal:** Go deep on P0 and P1 features. Spend time proportional to risk.
|
|
3
|
+
**Goal:** Go deep on P0 and P1 features. Spend time proportional to risk. Every command carries `-s=<sid>` (omitted here).
|
|
4
4
|
|
|
5
|
-
Focus on the features your risk ranking identified as P0/P1. Apply SFDIPOT, but only the
|
|
5
|
+
Focus on the features your risk ranking identified as P0/P1. Apply SFDIPOT, but only the
|
|
6
|
+
dimensions you selected in the charter.
|
|
6
7
|
|
|
7
8
|
## For Each P0/P1 Feature
|
|
8
9
|
|
|
@@ -21,55 +22,69 @@ playwright-cli fill <ref> "a" # Minimum
|
|
|
21
22
|
playwright-cli fill <ref> "<1000+ chars>" # Maximum
|
|
22
23
|
```
|
|
23
24
|
|
|
24
|
-
### Business Logic Stress (from knowledge base)
|
|
25
|
+
### Business Logic Stress (from the knowledge base)
|
|
25
26
|
|
|
26
|
-
- **Wrong state order:**
|
|
27
|
-
- **Parameter tampering:**
|
|
27
|
+
- **Wrong state order:** skip a step in a multi-step flow; navigate directly to confirmation
|
|
28
|
+
- **Parameter tampering:** change values that should be server-controlled
|
|
28
29
|
```bash
|
|
30
|
+
playwright-cli fill <ref> "-100" # preferred: types, so framework handlers fire
|
|
29
31
|
playwright-cli eval "document.querySelector('input[name=amount]').value = '-100'"
|
|
30
32
|
```
|
|
31
|
-
|
|
32
|
-
-
|
|
33
|
+
The `eval` form assigns `.value` without dispatching `input`/`change`, so a
|
|
34
|
+
React/Vue-controlled field reverts on the next render — prefer `fill`, and tamper at the
|
|
35
|
+
network layer (`route`) when the value is genuinely server-controlled.
|
|
36
|
+
- **Replay:** complete an action, then try to complete it again immediately
|
|
37
|
+
- **Self-referential:** transfer from account A to account A; apply a discount code twice
|
|
33
38
|
|
|
34
39
|
### Negative Space Check
|
|
35
40
|
|
|
36
41
|
- For EACH form, ask: "What fields should be here that aren't?"
|
|
37
42
|
- For EACH flow, ask: "What steps are missing?"
|
|
38
|
-
- Compare against domain checklist
|
|
43
|
+
- Compare against the domain checklist (`<data>/domains/<domain>.yml`)
|
|
39
44
|
|
|
40
45
|
### Consistency Oracles (FEW HICCUPPS)
|
|
41
46
|
|
|
42
47
|
After testing each feature, explicitly ask:
|
|
43
|
-
- **Users:** "Would a real user expect this
|
|
48
|
+
- **Users:** "Would a real user expect this behaviour?"
|
|
44
49
|
- **Claims:** "Does this match what the app promises?"
|
|
45
50
|
- **Product:** "Is this consistent with how other features in the app work?"
|
|
46
51
|
- **Standards:** "Does this meet industry standards for this domain?"
|
|
47
52
|
|
|
48
|
-
If you notice an inconsistency (
|
|
53
|
+
If you notice an inconsistency ("Bill Pay validates but Transfer doesn't"), **follow that
|
|
54
|
+
thread**; don't move on. The inconsistency is a signal.
|
|
49
55
|
|
|
50
|
-
###
|
|
56
|
+
### Stable locators for the bug report
|
|
51
57
|
|
|
52
|
-
|
|
58
|
+
A reproduction step must name the element unambiguously. When a ref is ambiguous:
|
|
53
59
|
|
|
54
|
-
|
|
55
|
-
-
|
|
60
|
+
```bash
|
|
61
|
+
playwright-cli generate-locator <ref> # a Playwright locator for the element behind the ref
|
|
62
|
+
playwright-cli highlight <ref> # confirm visually which element you mean, then
|
|
63
|
+
playwright-cli highlight --hide
|
|
64
|
+
```
|
|
56
65
|
|
|
57
|
-
|
|
66
|
+
Paste the generated locator into the bug's steps. `page.pickLocator()` (Playwright 1.59)
|
|
67
|
+
exists for the same purpose but is **human-only**: it opens an interactive picker in a headed
|
|
68
|
+
browser and waits for a click, so it is not usable from an agent session.
|
|
58
69
|
|
|
59
70
|
### Test Agents Handoff (optional, Playwright 1.56+)
|
|
60
71
|
|
|
61
|
-
When feature testing produces a **reproducible** bug (you can trigger it twice in a row,
|
|
72
|
+
When feature testing produces a **reproducible** bug (you can trigger it twice in a row,
|
|
73
|
+
same steps, same result), you may hand it off after the session to the Playwright Test
|
|
74
|
+
Agents **generator** to convert the reproduction into a regression test:
|
|
62
75
|
|
|
63
|
-
1. Finish
|
|
64
|
-
2. If
|
|
65
|
-
3. The generator produces a `tests/repro
|
|
66
|
-
4. The **healer** agent
|
|
76
|
+
1. Finish the bug report with exact steps (phase 7).
|
|
77
|
+
2. If Step 6 of setup bootstrapped Test Agents, pass the bug report to the generator agent.
|
|
78
|
+
3. The generator produces a `tests/repro-BUG-NNN.spec.ts` you can commit alongside the bug report.
|
|
79
|
+
4. The **healer** agent later repairs the test if it breaks for non-bug reasons.
|
|
67
80
|
|
|
68
|
-
Full workflow in
|
|
81
|
+
Full workflow in `${CLAUDE_SKILL_DIR}/references/playwright-agents-integration.md`. This
|
|
82
|
+
handoff is **opt-in**; it does not replace the bug report and is skipped entirely when
|
|
83
|
+
`@playwright/test` is not installed.
|
|
69
84
|
|
|
70
85
|
## After Feature Testing
|
|
71
86
|
|
|
72
|
-
Write
|
|
87
|
+
Write `phase-5-features.md` (confidentiality header first). Include the reasoning for each test.
|
|
73
88
|
|
|
74
|
-
**Update progress:**
|
|
89
|
+
**Update progress:** set the features phase complete in `progress.json`. Append to `session-log.md`:
|
|
75
90
|
`[<timestamp>] [PHASE] Features complete — <N> P0 features tested, <N> P1 features tested, <N> inconsistencies found, <N> bugs so far`
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# Phase 6: Edge Cases, Security & Negative Testing (
|
|
1
|
+
# Phase 6: Edge Cases, Security & Negative Testing (6 min)
|
|
2
2
|
|
|
3
|
-
**Goal:** Saboteur tour
|
|
3
|
+
**Goal:** Saboteur tour: actively try to break things. Every command carries `-s=<sid>` (omitted here). In read-only mode, fill but never submit.
|
|
4
4
|
|
|
5
5
|
## Input Attacks (from error-guessing knowledge)
|
|
6
6
|
|
|
@@ -17,15 +17,21 @@ playwright-cli fill <ref> " "
|
|
|
17
17
|
playwright-cli fill <ref> "A repeated 5000 times..."
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
+
After an XSS payload, check whether it executed: `playwright-cli console error` plus
|
|
21
|
+
`playwright-cli run-code "async page => (await page.pageErrors()).length"`, and look for a
|
|
22
|
+
dialog (`dialog-dismiss` if one opened).
|
|
23
|
+
|
|
20
24
|
## Race Conditions (if applicable)
|
|
21
25
|
|
|
22
26
|
```bash
|
|
23
|
-
# Open a second session
|
|
24
|
-
playwright-cli -s
|
|
25
|
-
playwright-cli -s
|
|
26
|
-
# Set up identical action in both sessions
|
|
27
|
+
# Open a second, separately isolated session alongside the main one
|
|
28
|
+
playwright-cli -s=<sid>-race open <url>
|
|
29
|
+
playwright-cli -s=<sid>-race state-load .auth/<target>.json
|
|
30
|
+
# Set up the identical action in both sessions
|
|
27
31
|
# Submit both as close to simultaneously as possible
|
|
28
|
-
# Check for double-processing
|
|
32
|
+
# Check for double-processing, then close the second session:
|
|
33
|
+
playwright-cli -s=<sid>-race close
|
|
34
|
+
playwright-cli -s=<sid>-race delete-data
|
|
29
35
|
```
|
|
30
36
|
|
|
31
37
|
## State Manipulation
|
|
@@ -33,14 +39,15 @@ playwright-cli -s=race state-load .auth/<target>.json
|
|
|
33
39
|
- Navigate directly to pages that require prior steps
|
|
34
40
|
- Use browser back/forward mid-flow
|
|
35
41
|
- Double-click submit buttons
|
|
36
|
-
- Simulate network errors: `playwright-cli route "**/*" --status=500`
|
|
42
|
+
- Simulate network errors: `playwright-cli route "**/*" --status=500` (then `playwright-cli unroute` to restore)
|
|
43
|
+
- Simulate offline: `playwright-cli network-state-set offline` / `online`
|
|
37
44
|
|
|
38
45
|
## Security
|
|
39
46
|
|
|
40
47
|
- Check for session tokens in URLs
|
|
41
48
|
- Try accessing admin/internal pages directly
|
|
42
|
-
- Check
|
|
43
|
-
- Test brute force: 5+ failed login attempts
|
|
49
|
+
- Check whether authenticated endpoints work without auth (`playwright-cli -s=<sid>-anon open <api-url>` in a fresh session, then close and `delete-data` it)
|
|
50
|
+
- Test brute force: 5+ failed login attempts; check for lockout
|
|
44
51
|
|
|
45
52
|
## Accessibility (quick check)
|
|
46
53
|
|
|
@@ -54,7 +61,11 @@ playwright-cli snapshot
|
|
|
54
61
|
playwright-cli resize 1280 720 # restore
|
|
55
62
|
```
|
|
56
63
|
|
|
57
|
-
|
|
64
|
+
The CLI `snapshot` **is** an aria snapshot: roles, names and states as assistive technology
|
|
65
|
+
sees them, so missing names and duplicate landmarks show up directly. For the raw YAML aria
|
|
66
|
+
tree of a region, `playwright-cli run-code "async page => page.locator('main').ariaSnapshot()"`.
|
|
67
|
+
Note: Playwright **removed** `page.accessibility` in 1.57; anything still calling it will
|
|
68
|
+
throw.
|
|
58
69
|
|
|
59
70
|
## Empty States
|
|
60
71
|
|
|
@@ -65,7 +76,7 @@ playwright-cli resize 1280 720 # restore
|
|
|
65
76
|
|
|
66
77
|
## After Edge Cases
|
|
67
78
|
|
|
68
|
-
Write
|
|
79
|
+
Write `phase-6-edge-cases.md` (confidentiality header first).
|
|
69
80
|
|
|
70
|
-
**Update progress:**
|
|
81
|
+
**Update progress:** set the edge_cases phase complete in `progress.json`. Append to `session-log.md`:
|
|
71
82
|
`[<timestamp>] [PHASE] Edge cases complete — <N> input attacks, <N> security checks, <N> a11y checks, <N> bugs so far`
|