qualiow-exploratory-testing 1.0.0 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "qualiow-exploratory-testing",
3
- "version": "1.0.0",
3
+ "version": "1.1.0",
4
4
  "description": "qualiow — AI-powered exploratory testing skills using Claude Code + Playwright CLI",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
@@ -19,6 +19,8 @@
19
19
  "docs/GETTING-STARTED.md"
20
20
  ],
21
21
  "scripts": {
22
+ "sync:skills": "rm -rf skills && cp -R .claude/skills skills",
23
+ "prebuild": "npm run sync:skills",
22
24
  "build": "tsup",
23
25
  "dev": "tsup --watch",
24
26
  "test": "vitest run",
@@ -53,7 +55,7 @@
53
55
  "url": "https://github.com/willcoliveira/qualiow-exploratory-testing-skills/issues"
54
56
  },
55
57
  "dependencies": {
56
- "@playwright/cli": "latest",
58
+ "@playwright/cli": "^0.1.6",
57
59
  "chalk": "^5.3.0",
58
60
  "commander": "^13.1.0",
59
61
  "glob": "^11.0.0",
@@ -61,6 +63,14 @@
61
63
  "yaml": "^2.7.0",
62
64
  "zod": "^3.24.0"
63
65
  },
66
+ "peerDependencies": {
67
+ "@playwright/test": "^1.59.1"
68
+ },
69
+ "peerDependenciesMeta": {
70
+ "@playwright/test": {
71
+ "optional": true
72
+ }
73
+ },
64
74
  "devDependencies": {
65
75
  "@types/node": "^22.0.0",
66
76
  "tsup": "^8.4.0",
@@ -7,7 +7,7 @@ description: >
7
7
  and writes bug reports that executives act on.
8
8
  Use when user says: "explore", "test this site", "find bugs", "QA check", "exploratory session",
9
9
  or provides a URL to test.
10
- allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Read, Write, Glob, Grep
10
+ allowed-tools: Bash(playwright-cli:*), Bash(npx playwright-cli:*), Bash(npx playwright:*), Read, Write, Glob, Grep
11
11
  ---
12
12
 
13
13
  # Exploratory Testing Session (v2)
@@ -68,3 +68,4 @@ Consult these as needed throughout the session:
68
68
  - **`references/severity-guide.md`** — Severity definitions (Critical/High/Medium/Low) and the "when in doubt go LOWER" rule
69
69
  - **`references/session-rules.md`** — 20 testing discipline rules: 45-min cap, one bug one report, evidence, risk-proportional time, adapt mid-session, log WHY, AI bias avoidance
70
70
  - **`references/context-integration.md`** — How `--context` changes the session (context file, inline context, blind mode)
71
+ - **`references/playwright-agents-integration.md`** — Opt-in handoff to Playwright Test Agents (planner / generator / healer, 1.56+) when a reproducible bug should become a regression test. Requires the `@playwright/test` peer dependency.
@@ -87,6 +87,20 @@ Append to `session-log.md`:
87
87
  [<timestamp>] [PHASE] Setup complete — session initialized
88
88
  ```
89
89
 
90
+ ## Step 6: Optional — Bootstrap Playwright Test Agents (opt-in)
91
+
92
+ **Skip this step entirely if `@playwright/test` is not installed as a peer dep OR `playwright.config.*` is not present.** This is an additive capability for consumers who want to pair an exploratory session with the Playwright Test Agents framework (planner / generator / healer, Playwright 1.56+).
93
+
94
+ If both prerequisites are present AND the user has opted in (via target config or inline context), you MAY offer to scaffold the agent workspace:
95
+
96
+ ```bash
97
+ npx playwright init-agents --loop=claude
98
+ ```
99
+
100
+ This generates agent definition files that later phases can hand off to — most usefully in phase 5 (features), where a reproducible bug can be converted into a regression test by the generator agent. See `references/playwright-agents-integration.md` for the full workflow.
101
+
102
+ **Never block the session on this step.** If the user declines, if the prerequisites are missing, or if the command fails, log a note in `session-log.md` and continue to Phase 1 normally.
103
+
90
104
  **IMPORTANT: After EVERY phase completion, update progress.json and append to session-log.md.** This is how the user monitors session progress. Use this pattern at the end of each phase:
91
105
 
92
106
  ```
@@ -37,6 +37,20 @@ On every page during discovery:
37
37
  - `playwright-cli network` -- note any failed requests (4xx, 5xx)
38
38
  - Watch for mixed content warnings (HTTP resources on HTTPS pages)
39
39
 
40
+ ### API-level observation helpers (optional, Playwright 1.56+)
41
+
42
+ If the session has access to the `@playwright/test` peer dep AND is running inside a Playwright test context (rare for qa-explore, common for bug-reproduction follow-ups), the following 1.56 retrieval methods can be used alongside the CLI commands above:
43
+
44
+ - `await page.consoleMessages()` — returns all console messages captured so far on the page. Useful when you need the full history rather than the most-recent tail.
45
+ - `await page.pageErrors()` — returns runtime page errors (uncaught exceptions). More reliable than filtering `console error` output.
46
+ - `await page.requests()` — returns all network requests issued by the page, with status, resource type, and timing. Use this for a single network snapshot instead of parsing `playwright-cli network` output twice.
47
+
48
+ These are OPTIONAL and only apply when exploratory testing is done from a Playwright test harness. The bash `playwright-cli` commands above remain the primary path for the skill.
49
+
50
+ ### Service Workers (Playwright 1.56+)
51
+
52
+ Playwright 1.56 added Service Worker network routing. If `playwright-cli network` shows requests you cannot account for on the main document, assume a Service Worker is intercepting them and flag this as a high-risk area worth re-testing with the route filter.
53
+
40
54
  ## After Discovery
41
55
 
42
56
  1. Write findings to `phase-1-discovery.md`
@@ -47,6 +47,26 @@ After testing each feature, explicitly ask:
47
47
 
48
48
  If you notice an inconsistency (e.g., "Bill Pay validates but Transfer doesn't"), **follow that thread** -- don't move on. The inconsistency is a signal.
49
49
 
50
+ ### Locator Stabilization Helpers (Playwright 1.59+, optional)
51
+
52
+ When testing a feature produces flaky snapshots because a locator isn't uniquely addressable, the following 1.59 helpers can harden the repro before you file a bug. They require the `@playwright/test` peer dep.
53
+
54
+ - `page.pickLocator()` — interactive locator picker. Use this when your current locator matches multiple elements and you need to find the one the user actually sees.
55
+ - `locator.normalize()` — returns a cleaned-up, canonical form of a locator. Use before copying a locator into a bug report so the reproduction is portable across runs.
56
+
57
+ These are diagnostic aids, not required steps. The bash `playwright-cli` flow above is the primary path.
58
+
59
+ ### Test Agents Handoff (optional, Playwright 1.56+)
60
+
61
+ When feature testing produces a **reproducible** bug (you can trigger it twice in a row, same steps, same result), you may hand off to the Playwright Test Agents **generator** to convert the reproduction into a regression test case:
62
+
63
+ 1. Finish writing the bug report with exact steps (see phase 07).
64
+ 2. If the session's Step 6 bootstrapped Test Agents, pass the repro steps to the generator agent.
65
+ 3. The generator produces a `tests/repro-<bug-id>.spec.ts` you can commit alongside the bug report.
66
+ 4. The **healer** agent can later be used by the team if the test becomes flaky.
67
+
68
+ Full workflow in `../references/playwright-agents-integration.md`. This handoff is **opt-in** — it does not replace the bug report and is skipped entirely when `@playwright/test` is not installed.
69
+
50
70
  ## After Feature Testing
51
71
 
52
72
  Write to `phase-3-feature-testing.md`. Include reasoning for each test.
@@ -54,6 +54,8 @@ playwright-cli snapshot
54
54
  playwright-cli resize 1280 720 # restore
55
55
  ```
56
56
 
57
+ > **Note on `page.accessibility` (Playwright 1.57 deprecation):** Upstream Playwright deprecated the `page.accessibility` snapshot API in 1.57. The replacement, available as of Playwright 1.59, is `page.ariaSnapshot()` — an aria-tree-based accessibility snapshot that is both more accurate and directly usable as a locator base. If a session upgrades to deeper a11y checks than the quick-pass above, prefer `page.ariaSnapshot()` over `page.accessibility`. The CLI checks above are unaffected and remain the primary path. See `../references/playwright-agents-integration.md` for a feature cross-reference.
58
+
57
59
  ## Empty States
58
60
 
59
61
  - What does the cart look like with no items?
@@ -7,6 +7,17 @@ playwright-cli tracing-stop
7
7
  playwright-cli close
8
8
  ```
9
9
 
10
+ ### Trace Analysis (Playwright 1.58–1.59, optional)
11
+
12
+ Once tracing is stopped, the captured trace can be explored from the command line or the HTML reporter:
13
+
14
+ - **`npx playwright trace <trace.zip>`** *(Playwright 1.59)* — opens the trace from the CLI without launching the full UI reporter. Useful for agent-driven debugging of a specific step or failure.
15
+ - **HTML reporter "Speedboard" timeline** *(Playwright 1.58)* — when the session generated a Playwright HTML report, the "Speedboard" tab shows a timeline visualization of steps useful for spotting latency anomalies that correlate with the bugs you filed.
16
+ - **Trace Viewer themes** *(Playwright 1.58)* — the Trace Viewer now honors system theme; no action required, just mentioned here so the reviewer knows the option exists.
17
+ - **`page.screencast()`** *(Playwright 1.59, optional)* — if the session wants to attach an annotated video of the reproduction to the bug report, `page.screencast()` captures it with action annotations and chapter overlays. Treat it as premium evidence for Critical / High bugs; a screenshot remains sufficient for Medium / Low.
18
+
19
+ These helpers are additive — the existing `screenshots/bug-NNN.png` workflow below is unchanged.
20
+
10
21
  ## Reflect Before Writing
11
22
 
12
23
  Before writing the report, answer these questions:
@@ -0,0 +1,126 @@
1
+ # Playwright Test Agents Integration (opt-in)
2
+
3
+ > **This file is opt-in reference material.** Read it only when the session has access to `@playwright/test` and the consumer project has a `playwright.config.*` file. Otherwise skip this file — `qa-explore` works end-to-end without it.
4
+
5
+ ## Purpose
6
+
7
+ `qa-explore` is an **exploratory** skill: its job is blind (or context-enriched) discovery of bugs, risks, and missing functionality in a web application, bounded by a 45-minute session.
8
+
9
+ **Playwright Test Agents** (introduced in Playwright 1.56) is a different, complementary capability: a set of three LLM-driven agents — **planner**, **generator**, **healer** — that author and maintain executable Playwright test specs.
10
+
11
+ These two are complementary, not overlapping:
12
+
13
+ | Concern | `qa-explore` | Playwright Test Agents |
14
+ |---|---|---|
15
+ | Goal | Find unknown bugs and risks | Generate and maintain regression tests |
16
+ | Output | Bug reports, session report, coverage map | `tests/*.spec.ts` files |
17
+ | When to use | New feature, new app, post-incident, pre-release audit | Known behavior you want locked in with a test |
18
+ | Input required | Just a URL (context optional) | A plan / spec / reproducible behavior |
19
+
20
+ Use `qa-explore` to discover; use Test Agents to codify a discovery into a regression test.
21
+
22
+ ## When to use this handoff
23
+
24
+ You are in the right file if ANY of these are true:
25
+
26
+ - Phase 5 or 6 of `qa-explore` produced a **reproducible** bug that deserves a regression test.
27
+ - The user completed a `qa-explore` session and explicitly asked "can you write a test for this bug?"
28
+ - The user wants to seed a new test project from a qa-explore charter.
29
+
30
+ You are in the WRONG file if:
31
+
32
+ - You are mid-session. Finish `qa-explore` first; Test Agents run after reporting.
33
+ - `@playwright/test` is not a resolved dependency in the consumer project (the peer dep is optional — consumers who didn't install it cannot use this file).
34
+ - The bug is not reproducible. Test Agents need deterministic input; flaky observations go in the bug report only.
35
+
36
+ ## Prerequisites
37
+
38
+ 1. `@playwright/test ^1.59.1` installed in the consumer project (declared as optional peer dep in this package).
39
+ 2. A `playwright.config.ts` or `playwright.config.js` present in the consumer project. If absent, run `npm init playwright@latest` first.
40
+ 3. The `qa-explore` session has completed at least Phase 5 (features) and produced a bug report with exact reproduction steps.
41
+
42
+ ## Bootstrap
43
+
44
+ From the consumer project root:
45
+
46
+ ```bash
47
+ npx playwright init-agents --loop=claude
48
+ ```
49
+
50
+ This generates the following (as of Playwright 1.59):
51
+
52
+ - Agent definition files for the selected loop (`claude`, `vscode`, or `opencode` are supported)
53
+ - A minimal `tests/agents/` scaffolding directory
54
+ - Updates to `playwright.config.ts` to wire agent reporters
55
+
56
+ The `--loop=claude` flag optimizes the agent definitions for Claude Code. `--loop=vscode` optimizes them for VS Code's built-in Copilot integration. Pick the one that matches the consumer's environment.
57
+
58
+ This command is idempotent — re-running it updates definitions without deleting committed tests.
59
+
60
+ ## The three agents
61
+
62
+ ### Planner
63
+
64
+ **Role:** Given a description of desired behavior (from a charter, a PRD, or a bug repro), produce an ordered plan of tests to write.
65
+
66
+ **Hand-off from qa-explore:** After a session, if the charter identified P0 journeys that have NO automated coverage, feed the charter and coverage map to the planner. The output is a list of test files the team should author.
67
+
68
+ **Output:** Markdown plan with one bullet per proposed test spec.
69
+
70
+ ### Generator
71
+
72
+ **Role:** Given a plan item (or a single reproducible bug), emit a working `.spec.ts` file.
73
+
74
+ **Hand-off from qa-explore:** This is the most common handoff. When Phase 5 or 6 produces a reproducible bug, pass the bug's repro steps + expected vs actual to the generator. The generator writes `tests/repro-<bug-id>.spec.ts`, typically a 20-40 line spec.
75
+
76
+ **Output:** A test spec file that fails initially (since the bug is not yet fixed) and will pass once the fix lands.
77
+
78
+ ### Healer
79
+
80
+ **Role:** When an existing spec becomes flaky — for example, because a locator shifted — the healer rewrites the failing parts of the spec to match the current DOM, using `locator.normalize()` and `page.pickLocator()` (Playwright 1.59) under the hood.
81
+
82
+ **Hand-off from qa-explore:** Not a direct handoff. Used by the consumer's team later when a generated spec starts failing due to non-bug changes.
83
+
84
+ ## Bug → regression test workflow
85
+
86
+ 1. **In Phase 5 or 6 of `qa-explore`**, confirm the bug is reproducible: trigger it twice in a row with the same steps and record identical results.
87
+ 2. **In Phase 7 (reporting)**, write the bug report with exact steps-to-reproduce and business impact. Place the generated `bugs/BUG-NNN.md` in the session output directory as usual.
88
+ 3. **After `qa-explore` finishes**, invoke the generator agent (via the loop configured at bootstrap) with the bug-report markdown as input. The generator produces `tests/repro-BUG-NNN.spec.ts`.
89
+ 4. **Run the new spec once** with `npx playwright test tests/repro-BUG-NNN.spec.ts` to confirm it fails (the bug is still present). Commit the failing test alongside the bug report; when the fix lands, the same test will pass.
90
+
91
+ This workflow treats `qa-explore` as the discovery phase and Test Agents as the codification phase.
92
+
93
+ ## Complementary tooling note
94
+
95
+ The official **Playwright MCP server** (see `github.com/microsoft/playwright-mcp`) is an alternative runtime harness for LLM-driven browser automation via accessibility trees. It overlaps with `playwright-cli` but targets long-lived agent sessions with persistent browser state. It is NOT required for Test Agents or `qa-explore` — mentioned here only so the consumer knows it exists.
96
+
97
+ ## Limitations
98
+
99
+ - **Opt-in only.** This file, the `Bash(npx playwright:*)` permission, and the `@playwright/test` peer dep are all optional. Consumers on Playwright 1.0–1.55 or without `@playwright/test` are unaffected — `qa-explore` works identically for them.
100
+ - **Not a replacement for the charter.** Test Agents turn known behavior into specs. They do not replace exploratory discovery; a planner fed only a charter will miss what the charter itself missed.
101
+ - **Requires a reproducible bug.** Flaky observations, race conditions that only triggered once, and "it felt wrong" findings are valuable in the bug report but cannot be codified by the generator.
102
+ - **Peer dep floor is 1.59.1.** This is deliberate: the APIs referenced here (`ariaSnapshot`, `screencast`, `pickLocator`, `normalize`, `trace` CLI) are all 1.59 features. Earlier Playwright versions will either omit these methods or expose older variants that do not match the guidance below.
103
+
104
+ ## Feature cross-reference
105
+
106
+ Playwright 1.56–1.59 additions and the qa-explore files that reference them:
107
+
108
+ | Feature | Playwright version | Referenced in |
109
+ |---|---|---|
110
+ | Test Agents framework (planner / generator / healer) | 1.56 | `SKILL.md`, `phases/00-setup.md`, `phases/05-features.md`, this file |
111
+ | `npx playwright init-agents` | 1.56 | `phases/00-setup.md`, `phases/05-features.md`, this file |
112
+ | `page.consoleMessages()` | 1.56 | `phases/03-discovery.md` |
113
+ | `page.pageErrors()` | 1.56 | `phases/03-discovery.md` |
114
+ | `page.requests()` | 1.56 | `phases/03-discovery.md` |
115
+ | Service Worker network routing | 1.56 | `phases/03-discovery.md` |
116
+ | `--test-list` / `--test-list-invert` | 1.56 | (used by Test Agents internally) |
117
+ | `page.accessibility` deprecation | 1.57 | `phases/06-edge-cases.md` |
118
+ | HTML reporter "Speedboard" timeline | 1.58 | `phases/07-reporting.md` |
119
+ | Trace Viewer themes | 1.58 | `phases/07-reporting.md` |
120
+ | `page.ariaSnapshot()` | 1.59 | `phases/06-edge-cases.md` |
121
+ | `page.pickLocator()` | 1.59 | `phases/05-features.md`, `qa-explore-quick/SKILL.md` |
122
+ | `locator.normalize()` | 1.59 | `phases/05-features.md`, `qa-explore-quick/SKILL.md` |
123
+ | `npx playwright trace` CLI | 1.59 | `phases/07-reporting.md` |
124
+ | `page.screencast()` | 1.59 | `phases/07-reporting.md` |
125
+ | `browser.bind()` | 1.59 | (reference only, not in phase guidance) |
126
+ | `--debug=cli` flag | 1.59 | (reference only, not in phase guidance) |
@@ -56,6 +56,7 @@ Based on what's on the page, apply relevant heuristics:
56
56
  - `playwright-cli network` — failed requests
57
57
  - `playwright-cli press Tab` (x5) — focus order and visibility
58
58
  - Responsive mobile: `playwright-cli resize 375 667`
59
+ - **Locator stability (optional, Playwright 1.59+):** if a target element is ambiguous, `page.pickLocator()` and `locator.normalize()` help produce a portable locator for the bug report. Only available when the consumer has installed the `@playwright/test` peer dep.
59
60
 
60
61
  For each bug: `playwright-cli screenshot` and document immediately.
61
62
 
@@ -104,3 +104,15 @@ Present the saved config to user:
104
104
  - Auth strategy configured
105
105
  - Scope defined
106
106
  - How to use: `/qa-explore --target <target-name>`
107
+
108
+ ### 9. Optional — Offer Playwright Test Agents Bootstrap
109
+
110
+ If the user plans to author full Playwright tests against this target later (not just exploratory sessions), offer to scaffold the Test Agents workspace now while you already have credentials and scope loaded:
111
+
112
+ ```bash
113
+ npx playwright init-agents --loop=claude
114
+ ```
115
+
116
+ This creates the planner / generator / healer agent definitions in the target project (Playwright 1.56+). It is **strictly optional** — declining does not affect the target config just written. If the user accepts, note in the target config's `notes` field that Test Agents are scaffolded; if they decline, continue normally.
117
+
118
+ This step requires `@playwright/test` to be installed in the target project. If it is not, skip this step silently.