qualiow-exploratory-testing 1.3.0 → 2.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (106) hide show
  1. package/.claude-plugin/marketplace.json +26 -0
  2. package/.claude-plugin/plugin.json +22 -0
  3. package/.env.example +27 -0
  4. package/CHANGELOG.md +259 -0
  5. package/README.md +472 -204
  6. package/agents/qa-diff-indexer-agent.md +81 -0
  7. package/agents/qa-gather-agent.md +41 -0
  8. package/agents/qa-page-mapper-agent.md +74 -0
  9. package/agents/qa-reporting-agent.md +103 -0
  10. package/{scripts → bin}/doctor-mobile.sh +57 -18
  11. package/bin/mcli +15 -9
  12. package/bin/mobile-cli.mjs +574 -286
  13. package/bin/qualiow +30 -0
  14. package/{scripts → bin}/setup-mobile.sh +56 -19
  15. package/bin/wadb +10 -4
  16. package/bin/wk-ios +29 -17
  17. package/bin/wkeval.mjs +7 -1
  18. package/data/domains/_default.yml +1 -1
  19. package/data/knowledge/changelog.yml +30 -12
  20. package/data/knowledge/manifest.yml +355 -127
  21. package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
  22. package/data/knowledge/releases/v0.1.0/release.yml +8 -1
  23. package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
  24. package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
  25. package/data/knowledge/releases/v0.6.0/release.yml +30 -0
  26. package/data/security/SECURITY-POLICY.md +273 -0
  27. package/data/targets/_default.yml +1 -1
  28. package/data/targets/_example-api-only.yml +80 -0
  29. package/data/targets/_example-mobile-emulation.yml +2 -2
  30. package/data/targets/_example-native-mobile.yml +2 -2
  31. package/data/targets/_example-sim-android-chrome.yml +1 -1
  32. package/data/targets/_example-sim-ios-safari.yml +1 -1
  33. package/data/templates/api-probe-matrix.md +3 -3
  34. package/data/templates/bug-report.md +29 -151
  35. package/data/templates/coverage-map.md +9 -6
  36. package/data/templates/expected-behaviour.md +3 -2
  37. package/data/templates/session-charter.md +3 -0
  38. package/data/templates/session-report.md +57 -80
  39. package/dist/cli/index.js +2286 -701
  40. package/dist/index.d.ts +637 -81
  41. package/dist/index.js +2058 -228
  42. package/docs/BACKEND-VERIFICATION.md +39 -9
  43. package/docs/GETTING-STARTED.md +300 -121
  44. package/docs/MOBILE-SETUP.md +149 -18
  45. package/hooks/hooks.json +27 -0
  46. package/hooks/scripts/read-guard.mjs +95 -0
  47. package/hooks/scripts/secret-patterns.d.mts +8 -0
  48. package/hooks/scripts/secret-patterns.mjs +136 -0
  49. package/hooks/scripts/write-guard.mjs +87 -0
  50. package/package.json +33 -18
  51. package/skills/qa-explore/SKILL.md +32 -25
  52. package/skills/qa-explore/phases/00-setup.md +76 -42
  53. package/skills/qa-explore/phases/01-auth.md +32 -15
  54. package/skills/qa-explore/phases/02-charter.md +16 -5
  55. package/skills/qa-explore/phases/03-discovery.md +93 -27
  56. package/skills/qa-explore/phases/04-journeys.md +2 -2
  57. package/skills/qa-explore/phases/05-features.md +39 -24
  58. package/skills/qa-explore/phases/06-edge-cases.md +24 -13
  59. package/skills/qa-explore/phases/07-reporting.md +102 -142
  60. package/skills/qa-explore/references/context-integration.md +1 -1
  61. package/skills/qa-explore/references/delegation-rules.md +82 -0
  62. package/skills/qa-explore/references/output-contract.md +192 -0
  63. package/skills/qa-explore/references/paths.md +108 -0
  64. package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
  65. package/skills/qa-explore/references/security-rules.md +70 -25
  66. package/skills/qa-explore/references/session-rules.md +2 -1
  67. package/skills/qa-explore/references/severity-guide.md +31 -4
  68. package/skills/qa-explore-cleanup/SKILL.md +51 -15
  69. package/skills/qa-explore-feedback/SKILL.md +23 -5
  70. package/skills/qa-explore-mobile/SKILL.md +86 -64
  71. package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
  72. package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
  73. package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
  74. package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
  75. package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
  76. package/skills/qa-explore-mobile/phases/05-features.md +9 -9
  77. package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
  78. package/skills/qa-explore-mobile/phases/07-reporting.md +108 -60
  79. package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
  80. package/skills/qa-explore-quick/SKILL.md +71 -26
  81. package/skills/qa-explore-report/SKILL.md +65 -35
  82. package/skills/qa-gather/SKILL.md +44 -8
  83. package/skills/qa-knowledge-add/SKILL.md +25 -9
  84. package/skills/qa-knowledge-list/SKILL.md +37 -40
  85. package/skills/qa-target-setup/SKILL.md +63 -41
  86. package/skills/qa-verify-backend/SKILL.md +31 -19
  87. package/skills/qa-verify-backend/phases/00-setup.md +57 -12
  88. package/skills/qa-verify-backend/phases/01-ac-decomposition.md +2 -2
  89. package/skills/qa-verify-backend/phases/02-static-review.md +11 -0
  90. package/skills/qa-verify-backend/phases/03-live-verification.md +5 -3
  91. package/skills/qa-verify-backend/phases/03b-api-verification.md +24 -9
  92. package/skills/qa-verify-backend/phases/04-e2e-trigger.md +18 -2
  93. package/skills/qa-verify-backend/phases/05-reporting.md +72 -12
  94. package/skills/qa-verify-backend/references/api-probes.md +28 -1
  95. package/skills/qa-verify-backend/references/payload-verification.md +1 -1
  96. package/skills/qa-verify-backend/references/release-readiness.md +1 -1
  97. package/skills/qa-verify-backend/references/safety-rules.md +13 -8
  98. package/data/domains/_default.md +0 -181
  99. package/data/domains/ecommerce.md +0 -192
  100. package/data/domains/fintech.md +0 -200
  101. package/data/domains/identity.md +0 -88
  102. package/data/domains/marketing.md +0 -194
  103. package/data/domains/saas.md +0 -197
  104. package/dist/cli/index.d.ts +0 -1
  105. package/dist/cli/index.js.map +0 -1
  106. package/dist/index.js.map +0 -1
@@ -4,7 +4,7 @@ How the `--context` parameter changes the session.
4
4
 
5
5
  ## Mode 1: Context File (`--context <file>`)
6
6
 
7
- Read the file. It may be a `tool-qa-workflow` output (structured markdown with Jira data, Confluence specs, GitLab MR diffs, Figma design notes) or any document describing what to test.
7
+ Read the file. It may be a `/qa-gather` context file (structured markdown with ticket data, specs, MR diffs, design notes) or any document describing what to test.
8
8
 
9
9
  Extract and use:
10
10
 
@@ -0,0 +1,82 @@
1
+ # Delegation Rules
2
+
3
+ What is allowed to leave this context, and what never is. The pack routes bulk input/output
4
+ away from the session and keeps the QA reasoning in it. Read this before reaching for a whole
5
+ file: most large reads have a cheaper route that returns the same facts.
6
+
7
+ ## Tiers
8
+
9
+ | Tier | What runs there | Use it for |
10
+ |---|---|---|
11
+ | **0 — deterministic code** | the `qualiow` CLI (`${CLAUDE_SKILL_DIR}/references/paths.md` resolves the binary) | anything with a fixed contract: knowledge digests, index rows, `stats.json` validation, the redaction scan, session listing and archival |
12
+ | **1 — cheap-model sub-agents** | `qa-reporting-agent` (sonnet), `qa-page-mapper-agent` (haiku), `qa-diff-indexer-agent` (haiku), `qa-gather-agent` (sonnet) | bounded reads that need light judgement and return a structured digest |
13
+ | **2 — this session** | the model reading these rules | exploration, interaction, bug finding, every judgement below |
14
+
15
+ Tier 0 beats tier 1 whenever the answer is deterministic: a command costs no tokens and
16
+ cannot hallucinate a row.
17
+
18
+ ## Delegates
19
+
20
+ Prefix every name with `qualiow:` under a plugin install (`qualiow:qa-reporting-agent`).
21
+
22
+ | Agent | Input | Returns | Used by |
23
+ |---|---|---|---|
24
+ | `qa-reporting-agent` | session directory + kind | `session-report.md` assembled from the notes, bugs and phase files, `qualiow session finalize` run, a summary of at most 8 lines | phase 7 of explore and mobile, phase 5 of backend, `/qa-explore-report` |
25
+ | `qa-page-mapper-agent` | a `snapshots/<page>.yml` over 300 lines + the page URL | forms with field refs, nav text → ref, interactive controls, visible error and empty-state text, hidden/disabled counts | phase 3 discovery |
26
+ | `qa-diff-indexer-agent` | repo, base, branch, the AC list | file → symbols → line ranges → candidate ACs, plus files matching no AC and ACs matching no file | phase 2 static review |
27
+ | `qa-gather-agent` | files, URLs or pasted text in the invocation | the context file under `output/context/`, gaps and assumptions marked | `/qa-gather` (always forks) |
28
+
29
+ If sub-agents are unavailable, do the step yourself as in 2.1.0 — the thresholds below still
30
+ apply; read in windows instead.
31
+
32
+ ## Never delegate
33
+
34
+ These stay in this session, always, whatever the tiering:
35
+
36
+ - **Severity and priority** — and the "when in doubt go LOWER" call
37
+ - **Business impact** — revenue, trust, regulatory, data, scale
38
+ - **Bug reports** — the decision that something *is* a bug, and every word of `bugs/BUG-NNN.md`
39
+ - **The charter** and the risk ranking (P0/P1/P2/P3) behind it
40
+ - **What's MISSING** — the negative-space question no extraction pass can ask
41
+ - **Verdicts** — `PASS`, `PARTIAL`, `FAIL`, `BLOCKED`, `NOT-REACHABLE`, `UNVERIFIABLE`
42
+ - **The executive summary** and the recommendations
43
+ - **The reflection** — what worried you, what you did not test, what to do next
44
+
45
+ A delegate that returns any of these has exceeded its brief; discard that part of its answer
46
+ and make the call yourself.
47
+
48
+ ## What a delegate must return
49
+
50
+ Bounded (say the maximum size in the request), structured (a table or a fixed heading set),
51
+ cited (`file:line`, a ref, or a probe command), and free of opinions — no "looks fine", no
52
+ "this is probably a bug", no severity. You then reason over its output; you never paste it
53
+ into a report unchanged.
54
+
55
+ ## Thresholds
56
+
57
+ | Input | Ceiling | Route instead |
58
+ |---|---|---|
59
+ | A qualiow data file (manifest, release entry, learned patterns) | 300 lines | `qualiow kb digest`, `qualiow list knowledge --entry <id>`, or `Grep` |
60
+ | A raw accessibility snapshot | 300 lines | `snapshot --depth=3`, `snapshot <ref>` to zoom, `find "<text>"` for one label |
61
+ | Any other file | 300 lines | `Read` with `offset`/`limit` after a `Grep` for the heading you need |
62
+ | A branch diff | 25 or more files, or 1,500 or more changed lines | `git diff --stat` first, then `git show <branch>:<path>` for the files that map to an AC |
63
+
64
+ Check before you read when you are unsure: `wc -l <file>`.
65
+
66
+ **Never `Read` `manifest.yml` or a knowledge release entry whole.** Use
67
+ `qualiow kb digest --for <explore|backend|mobile>` for the session's working set,
68
+ `qualiow list knowledge --entry <id>` for one entry in full, or `Grep` the release file for
69
+ the one heading you need.
70
+
71
+ ## Overrides
72
+
73
+ - `QUALIOW_READ_MAX_LINES=<n>` — moves the 300-line ceiling for a project
74
+ - `QUALIOW_HOOKS=off` — disables the enforcement layer for a project
75
+
76
+ Both are read by the hook layer — `hooks/hooks.json` in the plugin, or `qualiow init --hooks`
77
+ in an npm project — and belong in the `env` block of the project's `.claude/settings.json`.
78
+
79
+ - `CLAUDE_CODE_SUBAGENT_MODEL` — overrides the model of every sub-agent, whatever each agent
80
+ file pins
81
+ - `model: inherit` in a project's own copy of an agent file — that one delegate runs on the
82
+ session's model instead of the cheap one
@@ -0,0 +1,192 @@
1
+ # Output Contract
2
+
3
+ One format for every session kind (explore, quick, mobile, backend). The templates in
4
+ `<data>/templates/` follow it, `qualiow report` parses it, and the feedback and cleanup
5
+ skills read it. A session that deviates from it is invisible to all of them.
6
+
7
+ ## Confidentiality header
8
+
9
+ The first two lines of **every** markdown artefact (charter, phase files, bug reports,
10
+ session report, coverage map, AC matrix, expected-behaviour spec):
11
+
12
+ ```
13
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
14
+ > and application details. Do not share outside your organization without review.
15
+ ```
16
+
17
+ ## Session files
18
+
19
+ | File | Written by |
20
+ |---|---|
21
+ | `charter.md` | phase 2 |
22
+ | `session-log.md` | every phase, append-only |
23
+ | `progress.json` | phase 0, updated after every phase |
24
+ | `phase-3-discovery.md` … `phase-6-edge-cases.md` | phases 3–6 (file number = phase number) |
25
+ | `screenshots/BUG-NNN.png`, other `screenshots/*.png` | as taken |
26
+ | `videos/*.webm` (web) / `videos/*.mp4` (mobile) | recording |
27
+ | `snapshots/*.yml` | raw accessibility trees from `playwright-cli --raw snapshot`; working files, never shipped, and excluded from the secrets scan |
28
+ | `phase-7-notes.md` | phase 7 (backend: phase 5) — the session's own executive summary, coverage rows, observations, areas not tested, recommendations and reflection; the input the report is assembled from |
29
+ | `bugs/BUG-NNN.md` | phase 7 |
30
+ | `session-report.md` | the `qa-reporting-agent` sub-agent, from `phase-7-notes.md`, the bugs, `stats.json` and the phase files (quick sessions write it directly) |
31
+ | `stats.json` | phase 7 |
32
+ | mobile: `logs/BUG-NNN.log` · backend: `ac-matrix.md`, `evidence/`, `probes/`, `expected-behaviour.md` | as produced |
33
+
34
+ The coverage map lives inside `session-report.md`. There is no separate session-level
35
+ `coverage-map.md`; `/qa-explore-report` can export one on request.
36
+
37
+ ## Bug report — `bugs/BUG-NNN.md`
38
+
39
+ ```markdown
40
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
41
+ > and application details. Do not share outside your organization without review.
42
+
43
+ # BUG-NNN: [Component] fails [Condition] causing [Impact]
44
+
45
+ **Severity:** Critical | High | Medium | Low
46
+ **Priority:** P0 | P1 | P2 | P3
47
+ **Component:** <Component>
48
+ **URL:** <exact URL — mobile: screen name or deep link — backend: endpoint or resource>
49
+ **Environment:** <browser + viewport — mobile: mode / platform / device / OS — backend: env kind + build>
50
+ **Reproduction rate:** Always | Intermittent (~X%) | Once
51
+
52
+ ## Summary
53
+ <two sentences: what breaks and why it matters>
54
+
55
+ ## Expected Behavior
56
+ <from the user's perspective>
57
+
58
+ ## Actual Behavior
59
+ <be specific; quote error messages verbatim>
60
+
61
+ ## Steps to Reproduce
62
+ 1. <exact steps a developer can follow>
63
+
64
+ ## Business Impact
65
+ - **Revenue impact:** <or "none identified">
66
+ - **Trust impact:** …
67
+ - **Regulatory risk:** …
68
+ - **Data risk:** …
69
+ - **Scale:** <who is affected, under what conditions>
70
+
71
+ ## Evidence
72
+ - Screenshot: `screenshots/BUG-NNN.png`
73
+ - Video: `videos/<file>` (if recorded)
74
+ - Log: `logs/BUG-NNN.log` (mobile) / `evidence/<file>` (backend)
75
+ - Console errors: <or "none">
76
+ - Network failures: <or "none">
77
+
78
+ ## Recommended Fix Priority
79
+ <why this priority, relative to the other findings>
80
+ ```
81
+
82
+ Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is mandatory
83
+ (answer at least one bullet, write "none identified" for the rest); severity per
84
+ `severity-guide.md`, and when in doubt go one level lower.
85
+
86
+ ## Session report — `session-report.md`
87
+
88
+ ```markdown
89
+ > CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
90
+ > and application details. Do not share outside your organization without review.
91
+
92
+ # Session Report — <target>
93
+
94
+ ## Session Metadata
95
+ | Field | Value |
96
+ |---|---|
97
+ | Session ID | <session-dir> |
98
+ | Kind | explore / quick / mobile / backend |
99
+ | Target | <target id or name> |
100
+ | URL | <base url> |
101
+ | Date | <YYYY-MM-DD> |
102
+ | Duration | <N> min |
103
+ | Charter | charter.md |
104
+
105
+ ## Executive Summary
106
+ <three sentences: what was tested, what was found, the biggest risk>
107
+
108
+ ## Summary Stats
109
+ | Metric | Count |
110
+ |---|---|
111
+ | Pages explored | |
112
+ | Actions performed | |
113
+ | Bugs — Critical | |
114
+ | Bugs — High | |
115
+ | Bugs — Medium | |
116
+ | Bugs — Low | |
117
+ | **Total bugs** | |
118
+
119
+ ## Coverage Map
120
+ | Area | Risk | Status | Bugs | Notes |
121
+ |---|---|---|---|---|
122
+ | <area> | P0 | tested | 2 | |
123
+ | <area> | P1 | partial | 1 | keyboard covered submit |
124
+ | <area> | P2 | not-tested | — | time ran out |
125
+
126
+ ## Bugs Found
127
+ | # | ID | Title | Severity | Report |
128
+ |---|---|---|---|---|
129
+ | 1 | BUG-001 | <title> | High | bugs/BUG-001.md |
130
+
131
+ ## Observations
132
+ ## Areas Not Tested
133
+ | Area | Reason |
134
+ |---|---|
135
+
136
+ ## Recommendations
137
+ ## Reflection
138
+ ## Session Stats
139
+ | Metric | Value |
140
+ |---|---|
141
+ ```
142
+
143
+ Coverage `Status` ∈ `tested` · `partial` · `not-tested` · `code-verified-only` (believed
144
+ correct from reading source, never observed: this is UNVERIFIABLE, not a pass, and must not
145
+ be skimmed as green).
146
+
147
+ Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
148
+ `## AC Matrix Summary` (all six verdicts counted) after the executive summary.
149
+
150
+ ## `stats.json`
151
+
152
+ ```json
153
+ {
154
+ "session_id": "<session-dir>",
155
+ "kind": "explore",
156
+ "target": "<target id or url>",
157
+ "date": "<YYYY-MM-DD>",
158
+ "duration_min": 42,
159
+ "bugs_found": 5,
160
+ "severity_counts": { "critical": 1, "high": 2, "medium": 2, "low": 0 },
161
+ "pages_explored": 14,
162
+ "domain": "ecommerce",
163
+ "started_at": "<ISO timestamp>",
164
+ "completed_at": "<ISO timestamp>",
165
+ "phases_completed": 8,
166
+ "total_phases": 8,
167
+ "coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5 },
168
+ "evidence": { "screenshots": 9, "console_errors_found": 2, "network_failures_found": 1 },
169
+ "areas_not_tested": ["settings — time ran out"],
170
+ "blocked_by": null
171
+ }
172
+ ```
173
+
174
+ The first eight keys are required; the rest are optional. No other top-level keys (the
175
+ schema is strict). Mobile puts its device block under `coverage.mobile`; backend puts the
176
+ verdict counts under `coverage.verdicts`.
177
+
178
+ ## Index rows
179
+
180
+ Every session ends by appending one row to `output/sessions/INDEX.md` and one row per bug
181
+ to `output/bugs/all-bugs.md`. Both rows are written by `qualiow session finalize <session-dir>`,
182
+ which refuses the session — exit 1, naming the file — when an artefact is missing the
183
+ confidentiality header, an unredacted secret survives the scan, or `stats.json` does not
184
+ conform. The column order (defined once in `paths.md`) is:
185
+
186
+ ```
187
+ | Date | Kind | Target | Bugs | Duration | Status | Report |
188
+ | ID | Session | Title | Severity | Status | Report |
189
+ ```
190
+
191
+ A session that skips these rows is invisible to `/qa-explore-report`,
192
+ `/qa-explore-feedback`, `/qa-explore-cleanup` and `qualiow list sessions`.
@@ -0,0 +1,108 @@
1
+ # Paths and Resolution Order
2
+
3
+ Every qualiow skill resolves its inputs in the same order, whichever way the skills were
4
+ installed (`qualiow init` copy into a project, a Claude Code plugin, or this repository
5
+ checked out). Resolve once in the setup phase, then **write the resolved literal prefix in
6
+ every command**. Never put a shell variable as the first token of a command line: Claude
7
+ Code's permission rules match on the first literal token, so a command that begins with a
8
+ variable expansion matches no rule and prompts every time. Write `qa/bin/mcli snapshot`,
9
+ not a variable holding that prefix.
10
+
11
+ ## Target config
12
+
13
+ | Input | Resolution order (first existing wins) |
14
+ |---|---|
15
+ | `--target <name>` | `$PWD/data/targets/<name>.yml` → `${CLAUDE_PLUGIN_ROOT}/data/targets/<name>.yml` |
16
+ | no `--target` | `$PWD/qa/target.yml` → `<data>/targets/_default.yml` |
17
+
18
+ `qa/target.yml` is the **project-local** target: it lives next to the code under test, is
19
+ written by `/qa-target-setup`, and is the right place for anything with internal hostnames.
20
+ `data/targets/` holds shared and example targets.
21
+
22
+ ## Data directory (`<data>`)
23
+
24
+ First existing of: `$PWD/data` → `${CLAUDE_PLUGIN_ROOT}/data` →
25
+ `${CLAUDE_SKILL_DIR}/../../data` → `${CLAUDE_SKILL_DIR}/../../../data`. The `qualiow` CLI
26
+ resolves the same order — an explicit `--data <dir>`, then `$PWD/data`, then
27
+ `$CLAUDE_PLUGIN_ROOT/data`, then the data shipped inside the package.
28
+
29
+ Read from `<data>`: `domains/<domain>.yml`, `knowledge/manifest.yml`,
30
+ `knowledge/releases/<version>/entries/*.yml`, `knowledge/learned-patterns.md`,
31
+ `templates/*.md`, `security/SECURITY-POLICY.md`.
32
+
33
+ **Writes** (learned patterns from `/qa-explore-feedback`, custom entries from
34
+ `/qa-knowledge-add`) always go to `$PWD/data/…`; create the directory if it is missing.
35
+ Never write into a plugin directory.
36
+
37
+ ## Credentials
38
+
39
+ `$PWD/qa/.env` → `$PWD/.env`. Target configs name env vars; the values live only in these
40
+ files. Read a value when you need it, pass it to the command, and never echo it into a
41
+ log, a report, or a command line that will be recorded.
42
+
43
+ ## Mobile driver
44
+
45
+ First existing of: `$PWD/qa/bin/mcli` → `$PWD/bin/mcli` → `mcli` on `PATH` →
46
+ `${CLAUDE_PLUGIN_ROOT}/bin/mcli` → `${CLAUDE_SKILL_DIR}/../../bin/mcli`. Same order for
47
+ `wadb`, `wk-ios` and `doctor-mobile.sh`. The mobile phase files show `qa/bin/mcli` (the
48
+ default after `qualiow init`); in a checkout of this repository it is `bin/mcli`.
49
+ Substitute the prefix you resolved.
50
+
51
+ ## CLI
52
+
53
+ `qualiow` is the deterministic half of the pack: knowledge digests, session finalization,
54
+ listings, archival. First working of:
55
+
56
+ 1. `qualiow` on `PATH` — a marketplace plugin install (its `bin/` is on `PATH`), or a global
57
+ npm install
58
+ 2. `npx -y -p qualiow-exploratory-testing qualiow` — a project that depends on the npm package
59
+ 3. `${CLAUDE_PLUGIN_ROOT}/bin/qualiow` — a plugin loaded with `--plugin-dir`
60
+ 4. `bin/qualiow` — this repository checked out
61
+
62
+ The npm package is **`qualiow-exploratory-testing`** and the binary it installs is `qualiow`.
63
+ Never write bare `npx qualiow`: that spelling resolves a package that does not exist.
64
+
65
+ Same rule as the mobile driver — resolve once in setup, then write the resolved literal
66
+ prefix as the first token of every command. The phase files show `qualiow <command>` for
67
+ brevity; substitute the prefix you resolved.
68
+
69
+ ## Output
70
+
71
+ Always `$PWD/output/`. Never write session output anywhere else.
72
+
73
+ ### Session directory
74
+
75
+ `output/sessions/<YYYY-MM-DD-HHmm>-<kind>-<slug>/`
76
+
77
+ - `kind` ∈ `explore` | `quick` | `mobile` | `backend`
78
+ - `slug` = target id without a leading `_`; mobile: `<target>-<ios|android>`; backend: the
79
+ ticket id or the target id; an ad-hoc URL: the hostname with dots replaced by dashes.
80
+ Lowercase `[a-z0-9-]`, at most 40 characters.
81
+ - Date first, so alphabetical order is chronological and "latest" is unambiguous.
82
+
83
+ `--session <dir>`: when `qualiow explore` (pre-flight) already created the directory, reuse
84
+ it instead of creating a second one.
85
+
86
+ ### playwright-cli session id
87
+
88
+ `-s=<kind>-<HHmm>-<slug>` (for example `-s=explore-1420-parabank`). Every `playwright-cli`
89
+ command in the session carries it; the phase files omit the prefix for brevity. The session
90
+ ends with `playwright-cli -s=<sid> close` and then `playwright-cli -s=<sid> delete-data`.
91
+
92
+ ### Index rows
93
+
94
+ `qualiow session finalize <session-dir>` writes both rows (and creates either index file if
95
+ it is missing). The formats below are what it appends — write them by hand only when the CLI
96
+ is unavailable.
97
+
98
+ Append to `output/sessions/INDEX.md` (columns `| Date | Kind | Target | Bugs | Duration | Status | Report |`):
99
+
100
+ ```
101
+ | <YYYY-MM-DD> | <kind> | <target> | <bugs> | <N> min | complete | <session-dir>/session-report.md |
102
+ ```
103
+
104
+ Append one row per bug to `output/bugs/all-bugs.md` (columns `| ID | Session | Title | Severity | Status | Report |`):
105
+
106
+ ```
107
+ | BUG-NNN | <session-dir> | <title> | <severity> | open | <session-dir>/bugs/BUG-NNN.md |
108
+ ```
@@ -1,43 +1,43 @@
1
1
  # Playwright Test Agents Integration (opt-in)
2
2
 
3
- > **This file is opt-in reference material.** Read it only when the session has access to `@playwright/test` and the consumer project has a `playwright.config.*` file. Otherwise skip this file — `qa-explore` works end-to-end without it.
3
+ > **This file is opt-in reference material.** Read it only when the consumer project has
4
+ > `@playwright/test` installed and a `playwright.config.*` file. Otherwise skip it;
5
+ > `qa-explore` works end-to-end without it.
4
6
 
5
7
  ## Purpose
6
8
 
7
- `qa-explore` is an **exploratory** skill: its job is blind (or context-enriched) discovery of bugs, risks, and missing functionality in a web application, bounded by a 45-minute session.
9
+ `qa-explore` is an **exploratory** skill: blind (or context-enriched) discovery of bugs,
10
+ risks and missing functionality in a web application, bounded by a 45-minute session.
8
11
 
9
- **Playwright Test Agents** (introduced in Playwright 1.56) is a different, complementary capability: a set of three LLM-driven agents — **planner**, **generator**, **healer** — that author and maintain executable Playwright test specs.
10
-
11
- These two are complementary, not overlapping:
12
+ **Playwright Test Agents** (Playwright 1.56+) are a different, complementary capability:
13
+ three agent definitions, **planner**, **generator** and **healer**, that author and maintain
14
+ executable Playwright test specs.
12
15
 
13
16
  | Concern | `qa-explore` | Playwright Test Agents |
14
17
  |---|---|---|
15
18
  | Goal | Find unknown bugs and risks | Generate and maintain regression tests |
16
19
  | Output | Bug reports, session report, coverage map | `tests/*.spec.ts` files |
17
- | When to use | New feature, new app, post-incident, pre-release audit | Known behavior you want locked in with a test |
18
- | Input required | Just a URL (context optional) | A plan / spec / reproducible behavior |
20
+ | When to use | New feature, new app, post-incident, pre-release audit | Known behaviour you want locked in with a test |
21
+ | Input required | Just a URL (context optional) | A plan, a spec, or a reproducible behaviour |
19
22
 
20
23
  Use `qa-explore` to discover; use Test Agents to codify a discovery into a regression test.
21
24
 
22
25
  ## When to use this handoff
23
26
 
24
- You are in the right file if ANY of these are true:
25
-
26
- - Phase 5 or 6 of `qa-explore` produced a **reproducible** bug that deserves a regression test.
27
- - The user completed a `qa-explore` session and explicitly asked "can you write a test for this bug?"
28
- - The user wants to seed a new test project from a qa-explore charter.
29
-
30
- You are in the WRONG file if:
27
+ - Phase 5 or 6 produced a **reproducible** bug that deserves a regression test.
28
+ - The user completed a session and asked "can you write a test for this bug?"
29
+ - The user wants to seed a new test project from a session charter.
31
30
 
32
- - You are mid-session. Finish `qa-explore` first; Test Agents run after reporting.
33
- - `@playwright/test` is not a resolved dependency in the consumer project (the peer dep is optional — consumers who didn't install it cannot use this file).
34
- - The bug is not reproducible. Test Agents need deterministic input; flaky observations go in the bug report only.
31
+ Not mid-session (finish `qa-explore` first), not without `@playwright/test`, and not for a
32
+ bug that is not reproducible (flaky observations belong in the bug report only).
35
33
 
36
34
  ## Prerequisites
37
35
 
38
- 1. `@playwright/test ^1.59.1` installed in the consumer project (declared as optional peer dep in this package).
39
- 2. A `playwright.config.ts` or `playwright.config.js` present in the consumer project. If absent, run `npm init playwright@latest` first.
40
- 3. The `qa-explore` session has completed at least Phase 5 (features) and produced a bug report with exact reproduction steps.
36
+ 1. `@playwright/test ^1.59.1` installed in the consumer project (an optional peer
37
+ dependency of this package).
38
+ 2. A `playwright.config.ts` or `playwright.config.js` in the consumer project. If absent,
39
+ run `npm init playwright@latest` first.
40
+ 3. A completed session with a bug report that has exact reproduction steps.
41
41
 
42
42
  ## Bootstrap
43
43
 
@@ -47,80 +47,57 @@ From the consumer project root:
47
47
  npx playwright init-agents --loop=claude
48
48
  ```
49
49
 
50
- This generates the following (as of Playwright 1.59):
51
-
52
- - Agent definition files for the selected loop (`claude`, `vscode`, or `opencode` are supported)
53
- - A minimal `tests/agents/` scaffolding directory
54
- - Updates to `playwright.config.ts` to wire agent reporters
55
-
56
- The `--loop=claude` flag optimizes the agent definitions for Claude Code. `--loop=vscode` optimizes them for VS Code's built-in Copilot integration. Pick the one that matches the consumer's environment.
57
-
58
- This command is idempotent — re-running it updates definitions without deleting committed tests.
59
-
60
- ## The three agents
61
-
62
- ### Planner
63
-
64
- **Role:** Given a description of desired behavior (from a charter, a PRD, or a bug repro), produce an ordered plan of tests to write.
65
-
66
- **Hand-off from qa-explore:** After a session, if the charter identified P0 journeys that have NO automated coverage, feed the charter and coverage map to the planner. The output is a list of test files the team should author.
50
+ Supported loops: `claude`, `codex`, `copilot`, `opencode`, `vscode`, `vscode-legacy`.
51
+ With `--loop=claude` it writes:
67
52
 
68
- **Output:** Markdown plan with one bullet per proposed test spec.
53
+ - `.claude/agents/` with one Markdown definition each for the planner, generator and healer
54
+ - `.mcp.json` registering the `playwright-test` server (`npx playwright run-test-mcp-server`)
55
+ - `specs/` with a `README.md`, and a default seed spec file if the project has none
56
+ (`--prompts` additionally writes `.claude/prompts/`)
69
57
 
70
- ### Generator
58
+ It does not create a `tests/agents/` directory and does not edit `playwright.config.*`.
59
+ The definitions are generated files: re-run the command after upgrading Playwright so the
60
+ agents pick up new tools and instructions.
71
61
 
72
- **Role:** Given a plan item (or a single reproducible bug), emit a working `.spec.ts` file.
62
+ ## The three agents (upstream definitions)
73
63
 
74
- **Hand-off from qa-explore:** This is the most common handoff. When Phase 5 or 6 produces a reproducible bug, pass the bug's repro steps + expected vs actual to the generator. The generator writes `tests/repro-<bug-id>.spec.ts`, typically a 20-40 line spec.
75
-
76
- **Output:** A test spec file that fails initially (since the bug is not yet fixed) and will pass once the fix lands.
77
-
78
- ### Healer
79
-
80
- **Role:** When an existing spec becomes flaky — for example, because a locator shifted — the healer rewrites the failing parts of the spec to match the current DOM, using `locator.normalize()` and `page.pickLocator()` (Playwright 1.59) under the hood.
81
-
82
- **Hand-off from qa-explore:** Not a direct handoff. Used by the consumer's team later when a generated spec starts failing due to non-bug changes.
64
+ - **Planner** explores the app and produces a Markdown test plan. Hand-off from
65
+ `qa-explore`: feed it the charter and the coverage map for P0 journeys that have no
66
+ automated coverage.
67
+ - **Generator** transforms the Markdown plan into Playwright Test files. Hand-off: pass a
68
+ reproducible bug's steps and expected-vs-actual; it writes `tests/repro-BUG-NNN.spec.ts`,
69
+ which fails until the fix lands.
70
+ - **Healer** executes the test suite and automatically repairs failing tests. No direct
71
+ hand-off; the team uses it later when a generated spec breaks for non-bug reasons.
83
72
 
84
73
  ## Bug → regression test workflow
85
74
 
86
- 1. **In Phase 5 or 6 of `qa-explore`**, confirm the bug is reproducible: trigger it twice in a row with the same steps and record identical results.
87
- 2. **In Phase 7 (reporting)**, write the bug report with exact steps-to-reproduce and business impact. Place the generated `bugs/BUG-NNN.md` in the session output directory as usual.
88
- 3. **After `qa-explore` finishes**, invoke the generator agent (via the loop configured at bootstrap) with the bug-report markdown as input. The generator produces `tests/repro-BUG-NNN.spec.ts`.
89
- 4. **Run the new spec once** with `npx playwright test tests/repro-BUG-NNN.spec.ts` to confirm it fails (the bug is still present). Commit the failing test alongside the bug report; when the fix lands, the same test will pass.
90
-
91
- This workflow treats `qa-explore` as the discovery phase and Test Agents as the codification phase.
92
-
93
- ## Complementary tooling note
94
-
95
- The official **Playwright MCP server** (see `github.com/microsoft/playwright-mcp`) is an alternative runtime harness for LLM-driven browser automation via accessibility trees. It overlaps with `playwright-cli` but targets long-lived agent sessions with persistent browser state. It is NOT required for Test Agents or `qa-explore` — mentioned here only so the consumer knows it exists.
75
+ 1. **In phase 5 or 6**, confirm the bug is reproducible: trigger it twice with the same
76
+ steps and identical results.
77
+ 2. **In phase 7**, write `bugs/BUG-NNN.md` per `output-contract.md`.
78
+ 3. **After the session**, invoke the generator agent with the bug report as input.
79
+ 4. **Run the new spec once**: `npx playwright test tests/repro-BUG-NNN.spec.ts`. It should
80
+ fail while the bug exists. Commit it with the bug report.
96
81
 
97
82
  ## Limitations
98
83
 
99
- - **Opt-in only.** This file, the `Bash(npx playwright:*)` permission, and the `@playwright/test` peer dep are all optional. Consumers on Playwright 1.0–1.55 or without `@playwright/test` are unaffected — `qa-explore` works identically for them.
100
- - **Not a replacement for the charter.** Test Agents turn known behavior into specs. They do not replace exploratory discovery; a planner fed only a charter will miss what the charter itself missed.
101
- - **Requires a reproducible bug.** Flaky observations, race conditions that only triggered once, and "it felt wrong" findings are valuable in the bug report but cannot be codified by the generator.
102
- - **Peer dep floor is 1.59.1.** This is deliberate: the APIs referenced here (`ariaSnapshot`, `screencast`, `pickLocator`, `normalize`, `trace` CLI) are all 1.59 features. Earlier Playwright versions will either omit these methods or expose older variants that do not match the guidance below.
84
+ - Opt-in only. This file, the `Bash(npx playwright:*)` permission and the peer dependency
85
+ are all optional; sessions that never touch them are unaffected.
86
+ - Test Agents turn known behaviour into specs. They do not replace exploratory discovery.
87
+ - The generator needs deterministic reproduction steps.
103
88
 
104
89
  ## Feature cross-reference
105
90
 
106
- Playwright 1.56–1.59 additions and the qa-explore files that reference them:
107
-
108
- | Feature | Playwright version | Referenced in |
109
- |---|---|---|
110
- | Test Agents framework (planner / generator / healer) | 1.56 | `SKILL.md`, `phases/00-setup.md`, `phases/05-features.md`, this file |
111
- | `npx playwright init-agents` | 1.56 | `phases/00-setup.md`, `phases/05-features.md`, this file |
112
- | `page.consoleMessages()` | 1.56 | `phases/03-discovery.md` |
113
- | `page.pageErrors()` | 1.56 | `phases/03-discovery.md` |
114
- | `page.requests()` | 1.56 | `phases/03-discovery.md` |
115
- | Service Worker network routing | 1.56 | `phases/03-discovery.md` |
116
- | `--test-list` / `--test-list-invert` | 1.56 | (used by Test Agents internally) |
117
- | `page.accessibility` deprecation | 1.57 | `phases/06-edge-cases.md` |
118
- | HTML reporter "Speedboard" timeline | 1.58 | `phases/07-reporting.md` |
119
- | Trace Viewer themes | 1.58 | `phases/07-reporting.md` |
120
- | `page.ariaSnapshot()` | 1.59 | `phases/06-edge-cases.md` |
121
- | `page.pickLocator()` | 1.59 | `phases/05-features.md`, `qa-explore-quick/SKILL.md` |
122
- | `locator.normalize()` | 1.59 | `phases/05-features.md`, `qa-explore-quick/SKILL.md` |
123
- | `npx playwright trace` CLI | 1.59 | `phases/07-reporting.md` |
124
- | `page.screencast()` | 1.59 | `phases/07-reporting.md` |
125
- | `browser.bind()` | 1.59 | (reference only, not in phase guidance) |
126
- | `--debug=cli` flag | 1.59 | (reference only, not in phase guidance) |
91
+ Playwright APIs the phases mention, with the version that introduced them, and how each is
92
+ reached from a `playwright-cli` session:
93
+
94
+ | Feature | Version | Used in | How |
95
+ |---|---|---|---|
96
+ | `npx playwright init-agents` | 1.56 | `${CLAUDE_SKILL_DIR}/phases/00-setup.md`, this file | shell |
97
+ | `page.consoleMessages()`, `page.pageErrors()`, `page.requests()` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/03-discovery.md` | `playwright-cli run-code "async page => …"` |
98
+ | `page.accessibility` **removed** | 1.57 | `${CLAUDE_SKILL_DIR}/phases/06-edge-cases.md` | use `snapshot` (aria tree) or `run-code` + `page.ariaSnapshot()` |
99
+ | `locator.ariaSnapshot()` (1.49), `page.ariaSnapshot()` (1.59) | 1.49 / 1.59 | `${CLAUDE_SKILL_DIR}/phases/06-edge-cases.md` | `run-code` |
100
+ | `page.pickLocator()` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/05-features.md` | human-only (opens a picker in a headed browser); agents use `generate-locator <ref>` |
101
+ | `page.screencast` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/07-reporting.md` | `video-start` / `video-chapter`, or a `run-code --filename` hero script |
102
+ | `npx playwright trace open <trace>` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/07-reporting.md` | shell |
103
+ | `--debug=cli`, `browser.bind()` | 1.59 | reference only | not used by the session |