qualiow-exploratory-testing 1.3.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +26 -0
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +259 -0
- package/README.md +472 -204
- package/agents/qa-diff-indexer-agent.md +81 -0
- package/agents/qa-gather-agent.md +41 -0
- package/agents/qa-page-mapper-agent.md +74 -0
- package/agents/qa-reporting-agent.md +103 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/bin/qualiow +30 -0
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/knowledge/changelog.yml +30 -12
- package/data/knowledge/manifest.yml +355 -127
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +3 -3
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +9 -6
- package/data/templates/expected-behaviour.md +3 -2
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +2286 -701
- package/dist/index.d.ts +637 -81
- package/dist/index.js +2058 -228
- package/docs/BACKEND-VERIFICATION.md +39 -9
- package/docs/GETTING-STARTED.md +300 -121
- package/docs/MOBILE-SETUP.md +149 -18
- package/hooks/hooks.json +27 -0
- package/hooks/scripts/read-guard.mjs +95 -0
- package/hooks/scripts/secret-patterns.d.mts +8 -0
- package/hooks/scripts/secret-patterns.mjs +136 -0
- package/hooks/scripts/write-guard.mjs +87 -0
- package/package.json +33 -18
- package/skills/qa-explore/SKILL.md +32 -25
- package/skills/qa-explore/phases/00-setup.md +76 -42
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +93 -27
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +102 -142
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/delegation-rules.md +82 -0
- package/skills/qa-explore/references/output-contract.md +192 -0
- package/skills/qa-explore/references/paths.md +108 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +70 -25
- package/skills/qa-explore/references/session-rules.md +2 -1
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +51 -15
- package/skills/qa-explore-feedback/SKILL.md +23 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +108 -60
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +71 -26
- package/skills/qa-explore-report/SKILL.md +65 -35
- package/skills/qa-gather/SKILL.md +44 -8
- package/skills/qa-knowledge-add/SKILL.md +25 -9
- package/skills/qa-knowledge-list/SKILL.md +37 -40
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +31 -19
- package/skills/qa-verify-backend/phases/00-setup.md +57 -12
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +2 -2
- package/skills/qa-verify-backend/phases/02-static-review.md +11 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +5 -3
- package/skills/qa-verify-backend/phases/03b-api-verification.md +24 -9
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +18 -2
- package/skills/qa-verify-backend/phases/05-reporting.md +72 -12
- package/skills/qa-verify-backend/references/api-probes.md +28 -1
- package/skills/qa-verify-backend/references/payload-verification.md +1 -1
- package/skills/qa-verify-backend/references/release-readiness.md +1 -1
- package/skills/qa-verify-backend/references/safety-rules.md +13 -8
- package/data/domains/_default.md +0 -181
- package/data/domains/ecommerce.md +0 -192
- package/data/domains/fintech.md +0 -200
- package/data/domains/identity.md +0 -88
- package/data/domains/marketing.md +0 -194
- package/data/domains/saas.md +0 -197
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
|
@@ -4,7 +4,7 @@ How the `--context` parameter changes the session.
|
|
|
4
4
|
|
|
5
5
|
## Mode 1: Context File (`--context <file>`)
|
|
6
6
|
|
|
7
|
-
Read the file. It may be a
|
|
7
|
+
Read the file. It may be a `/qa-gather` context file (structured markdown with ticket data, specs, MR diffs, design notes) or any document describing what to test.
|
|
8
8
|
|
|
9
9
|
Extract and use:
|
|
10
10
|
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Delegation Rules
|
|
2
|
+
|
|
3
|
+
What is allowed to leave this context, and what never is. The pack routes bulk input/output
|
|
4
|
+
away from the session and keeps the QA reasoning in it. Read this before reaching for a whole
|
|
5
|
+
file: most large reads have a cheaper route that returns the same facts.
|
|
6
|
+
|
|
7
|
+
## Tiers
|
|
8
|
+
|
|
9
|
+
| Tier | What runs there | Use it for |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| **0 — deterministic code** | the `qualiow` CLI (`${CLAUDE_SKILL_DIR}/references/paths.md` resolves the binary) | anything with a fixed contract: knowledge digests, index rows, `stats.json` validation, the redaction scan, session listing and archival |
|
|
12
|
+
| **1 — cheap-model sub-agents** | `qa-reporting-agent` (sonnet), `qa-page-mapper-agent` (haiku), `qa-diff-indexer-agent` (haiku), `qa-gather-agent` (sonnet) | bounded reads that need light judgement and return a structured digest |
|
|
13
|
+
| **2 — this session** | the model reading these rules | exploration, interaction, bug finding, every judgement below |
|
|
14
|
+
|
|
15
|
+
Tier 0 beats tier 1 whenever the answer is deterministic: a command costs no tokens and
|
|
16
|
+
cannot hallucinate a row.
|
|
17
|
+
|
|
18
|
+
## Delegates
|
|
19
|
+
|
|
20
|
+
Prefix every name with `qualiow:` under a plugin install (`qualiow:qa-reporting-agent`).
|
|
21
|
+
|
|
22
|
+
| Agent | Input | Returns | Used by |
|
|
23
|
+
|---|---|---|---|
|
|
24
|
+
| `qa-reporting-agent` | session directory + kind | `session-report.md` assembled from the notes, bugs and phase files, `qualiow session finalize` run, a summary of at most 8 lines | phase 7 of explore and mobile, phase 5 of backend, `/qa-explore-report` |
|
|
25
|
+
| `qa-page-mapper-agent` | a `snapshots/<page>.yml` over 300 lines + the page URL | forms with field refs, nav text → ref, interactive controls, visible error and empty-state text, hidden/disabled counts | phase 3 discovery |
|
|
26
|
+
| `qa-diff-indexer-agent` | repo, base, branch, the AC list | file → symbols → line ranges → candidate ACs, plus files matching no AC and ACs matching no file | phase 2 static review |
|
|
27
|
+
| `qa-gather-agent` | files, URLs or pasted text in the invocation | the context file under `output/context/`, gaps and assumptions marked | `/qa-gather` (always forks) |
|
|
28
|
+
|
|
29
|
+
If sub-agents are unavailable, do the step yourself as in 2.1.0 — the thresholds below still
|
|
30
|
+
apply; read in windows instead.
|
|
31
|
+
|
|
32
|
+
## Never delegate
|
|
33
|
+
|
|
34
|
+
These stay in this session, always, whatever the tiering:
|
|
35
|
+
|
|
36
|
+
- **Severity and priority** — and the "when in doubt go LOWER" call
|
|
37
|
+
- **Business impact** — revenue, trust, regulatory, data, scale
|
|
38
|
+
- **Bug reports** — the decision that something *is* a bug, and every word of `bugs/BUG-NNN.md`
|
|
39
|
+
- **The charter** and the risk ranking (P0/P1/P2/P3) behind it
|
|
40
|
+
- **What's MISSING** — the negative-space question no extraction pass can ask
|
|
41
|
+
- **Verdicts** — `PASS`, `PARTIAL`, `FAIL`, `BLOCKED`, `NOT-REACHABLE`, `UNVERIFIABLE`
|
|
42
|
+
- **The executive summary** and the recommendations
|
|
43
|
+
- **The reflection** — what worried you, what you did not test, what to do next
|
|
44
|
+
|
|
45
|
+
A delegate that returns any of these has exceeded its brief; discard that part of its answer
|
|
46
|
+
and make the call yourself.
|
|
47
|
+
|
|
48
|
+
## What a delegate must return
|
|
49
|
+
|
|
50
|
+
Bounded (say the maximum size in the request), structured (a table or a fixed heading set),
|
|
51
|
+
cited (`file:line`, a ref, or a probe command), and free of opinions — no "looks fine", no
|
|
52
|
+
"this is probably a bug", no severity. You then reason over its output; you never paste it
|
|
53
|
+
into a report unchanged.
|
|
54
|
+
|
|
55
|
+
## Thresholds
|
|
56
|
+
|
|
57
|
+
| Input | Ceiling | Route instead |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| A qualiow data file (manifest, release entry, learned patterns) | 300 lines | `qualiow kb digest`, `qualiow list knowledge --entry <id>`, or `Grep` |
|
|
60
|
+
| A raw accessibility snapshot | 300 lines | `snapshot --depth=3`, `snapshot <ref>` to zoom, `find "<text>"` for one label |
|
|
61
|
+
| Any other file | 300 lines | `Read` with `offset`/`limit` after a `Grep` for the heading you need |
|
|
62
|
+
| A branch diff | 25 or more files, or 1,500 or more changed lines | `git diff --stat` first, then `git show <branch>:<path>` for the files that map to an AC |
|
|
63
|
+
|
|
64
|
+
Check before you read when you are unsure: `wc -l <file>`.
|
|
65
|
+
|
|
66
|
+
**Never `Read` `manifest.yml` or a knowledge release entry whole.** Use
|
|
67
|
+
`qualiow kb digest --for <explore|backend|mobile>` for the session's working set,
|
|
68
|
+
`qualiow list knowledge --entry <id>` for one entry in full, or `Grep` the release file for
|
|
69
|
+
the one heading you need.
|
|
70
|
+
|
|
71
|
+
## Overrides
|
|
72
|
+
|
|
73
|
+
- `QUALIOW_READ_MAX_LINES=<n>` — moves the 300-line ceiling for a project
|
|
74
|
+
- `QUALIOW_HOOKS=off` — disables the enforcement layer for a project
|
|
75
|
+
|
|
76
|
+
Both are read by the hook layer — `hooks/hooks.json` in the plugin, or `qualiow init --hooks`
|
|
77
|
+
in an npm project — and belong in the `env` block of the project's `.claude/settings.json`.
|
|
78
|
+
|
|
79
|
+
- `CLAUDE_CODE_SUBAGENT_MODEL` — overrides the model of every sub-agent, whatever each agent
|
|
80
|
+
file pins
|
|
81
|
+
- `model: inherit` in a project's own copy of an agent file — that one delegate runs on the
|
|
82
|
+
session's model instead of the cheap one
|
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
# Output Contract
|
|
2
|
+
|
|
3
|
+
One format for every session kind (explore, quick, mobile, backend). The templates in
|
|
4
|
+
`<data>/templates/` follow it, `qualiow report` parses it, and the feedback and cleanup
|
|
5
|
+
skills read it. A session that deviates from it is invisible to all of them.
|
|
6
|
+
|
|
7
|
+
## Confidentiality header
|
|
8
|
+
|
|
9
|
+
The first two lines of **every** markdown artefact (charter, phase files, bug reports,
|
|
10
|
+
session report, coverage map, AC matrix, expected-behaviour spec):
|
|
11
|
+
|
|
12
|
+
```
|
|
13
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
14
|
+
> and application details. Do not share outside your organization without review.
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Session files
|
|
18
|
+
|
|
19
|
+
| File | Written by |
|
|
20
|
+
|---|---|
|
|
21
|
+
| `charter.md` | phase 2 |
|
|
22
|
+
| `session-log.md` | every phase, append-only |
|
|
23
|
+
| `progress.json` | phase 0, updated after every phase |
|
|
24
|
+
| `phase-3-discovery.md` … `phase-6-edge-cases.md` | phases 3–6 (file number = phase number) |
|
|
25
|
+
| `screenshots/BUG-NNN.png`, other `screenshots/*.png` | as taken |
|
|
26
|
+
| `videos/*.webm` (web) / `videos/*.mp4` (mobile) | recording |
|
|
27
|
+
| `snapshots/*.yml` | raw accessibility trees from `playwright-cli --raw snapshot`; working files, never shipped, and excluded from the secrets scan |
|
|
28
|
+
| `phase-7-notes.md` | phase 7 (backend: phase 5) — the session's own executive summary, coverage rows, observations, areas not tested, recommendations and reflection; the input the report is assembled from |
|
|
29
|
+
| `bugs/BUG-NNN.md` | phase 7 |
|
|
30
|
+
| `session-report.md` | the `qa-reporting-agent` sub-agent, from `phase-7-notes.md`, the bugs, `stats.json` and the phase files (quick sessions write it directly) |
|
|
31
|
+
| `stats.json` | phase 7 |
|
|
32
|
+
| mobile: `logs/BUG-NNN.log` · backend: `ac-matrix.md`, `evidence/`, `probes/`, `expected-behaviour.md` | as produced |
|
|
33
|
+
|
|
34
|
+
The coverage map lives inside `session-report.md`. There is no separate session-level
|
|
35
|
+
`coverage-map.md`; `/qa-explore-report` can export one on request.
|
|
36
|
+
|
|
37
|
+
## Bug report — `bugs/BUG-NNN.md`
|
|
38
|
+
|
|
39
|
+
```markdown
|
|
40
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
41
|
+
> and application details. Do not share outside your organization without review.
|
|
42
|
+
|
|
43
|
+
# BUG-NNN: [Component] fails [Condition] causing [Impact]
|
|
44
|
+
|
|
45
|
+
**Severity:** Critical | High | Medium | Low
|
|
46
|
+
**Priority:** P0 | P1 | P2 | P3
|
|
47
|
+
**Component:** <Component>
|
|
48
|
+
**URL:** <exact URL — mobile: screen name or deep link — backend: endpoint or resource>
|
|
49
|
+
**Environment:** <browser + viewport — mobile: mode / platform / device / OS — backend: env kind + build>
|
|
50
|
+
**Reproduction rate:** Always | Intermittent (~X%) | Once
|
|
51
|
+
|
|
52
|
+
## Summary
|
|
53
|
+
<two sentences: what breaks and why it matters>
|
|
54
|
+
|
|
55
|
+
## Expected Behavior
|
|
56
|
+
<from the user's perspective>
|
|
57
|
+
|
|
58
|
+
## Actual Behavior
|
|
59
|
+
<be specific; quote error messages verbatim>
|
|
60
|
+
|
|
61
|
+
## Steps to Reproduce
|
|
62
|
+
1. <exact steps a developer can follow>
|
|
63
|
+
|
|
64
|
+
## Business Impact
|
|
65
|
+
- **Revenue impact:** <or "none identified">
|
|
66
|
+
- **Trust impact:** …
|
|
67
|
+
- **Regulatory risk:** …
|
|
68
|
+
- **Data risk:** …
|
|
69
|
+
- **Scale:** <who is affected, under what conditions>
|
|
70
|
+
|
|
71
|
+
## Evidence
|
|
72
|
+
- Screenshot: `screenshots/BUG-NNN.png`
|
|
73
|
+
- Video: `videos/<file>` (if recorded)
|
|
74
|
+
- Log: `logs/BUG-NNN.log` (mobile) / `evidence/<file>` (backend)
|
|
75
|
+
- Console errors: <or "none">
|
|
76
|
+
- Network failures: <or "none">
|
|
77
|
+
|
|
78
|
+
## Recommended Fix Priority
|
|
79
|
+
<why this priority, relative to the other findings>
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Rules: numbered from `BUG-001` per session; one bug per file; Business Impact is mandatory
|
|
83
|
+
(answer at least one bullet, write "none identified" for the rest); severity per
|
|
84
|
+
`severity-guide.md`, and when in doubt go one level lower.
|
|
85
|
+
|
|
86
|
+
## Session report — `session-report.md`
|
|
87
|
+
|
|
88
|
+
```markdown
|
|
89
|
+
> CONFIDENTIAL: This report may contain internal URLs, security vulnerabilities,
|
|
90
|
+
> and application details. Do not share outside your organization without review.
|
|
91
|
+
|
|
92
|
+
# Session Report — <target>
|
|
93
|
+
|
|
94
|
+
## Session Metadata
|
|
95
|
+
| Field | Value |
|
|
96
|
+
|---|---|
|
|
97
|
+
| Session ID | <session-dir> |
|
|
98
|
+
| Kind | explore / quick / mobile / backend |
|
|
99
|
+
| Target | <target id or name> |
|
|
100
|
+
| URL | <base url> |
|
|
101
|
+
| Date | <YYYY-MM-DD> |
|
|
102
|
+
| Duration | <N> min |
|
|
103
|
+
| Charter | charter.md |
|
|
104
|
+
|
|
105
|
+
## Executive Summary
|
|
106
|
+
<three sentences: what was tested, what was found, the biggest risk>
|
|
107
|
+
|
|
108
|
+
## Summary Stats
|
|
109
|
+
| Metric | Count |
|
|
110
|
+
|---|---|
|
|
111
|
+
| Pages explored | |
|
|
112
|
+
| Actions performed | |
|
|
113
|
+
| Bugs — Critical | |
|
|
114
|
+
| Bugs — High | |
|
|
115
|
+
| Bugs — Medium | |
|
|
116
|
+
| Bugs — Low | |
|
|
117
|
+
| **Total bugs** | |
|
|
118
|
+
|
|
119
|
+
## Coverage Map
|
|
120
|
+
| Area | Risk | Status | Bugs | Notes |
|
|
121
|
+
|---|---|---|---|---|
|
|
122
|
+
| <area> | P0 | tested | 2 | |
|
|
123
|
+
| <area> | P1 | partial | 1 | keyboard covered submit |
|
|
124
|
+
| <area> | P2 | not-tested | — | time ran out |
|
|
125
|
+
|
|
126
|
+
## Bugs Found
|
|
127
|
+
| # | ID | Title | Severity | Report |
|
|
128
|
+
|---|---|---|---|---|
|
|
129
|
+
| 1 | BUG-001 | <title> | High | bugs/BUG-001.md |
|
|
130
|
+
|
|
131
|
+
## Observations
|
|
132
|
+
## Areas Not Tested
|
|
133
|
+
| Area | Reason |
|
|
134
|
+
|---|---|
|
|
135
|
+
|
|
136
|
+
## Recommendations
|
|
137
|
+
## Reflection
|
|
138
|
+
## Session Stats
|
|
139
|
+
| Metric | Value |
|
|
140
|
+
|---|---|
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Coverage `Status` ∈ `tested` · `partial` · `not-tested` · `code-verified-only` (believed
|
|
144
|
+
correct from reading source, never observed: this is UNVERIFIABLE, not a pass, and must not
|
|
145
|
+
be skimmed as green).
|
|
146
|
+
|
|
147
|
+
Mobile appends `## Mobile Context` and `## Deferred Tests`; backend inserts
|
|
148
|
+
`## AC Matrix Summary` (all six verdicts counted) after the executive summary.
|
|
149
|
+
|
|
150
|
+
## `stats.json`
|
|
151
|
+
|
|
152
|
+
```json
|
|
153
|
+
{
|
|
154
|
+
"session_id": "<session-dir>",
|
|
155
|
+
"kind": "explore",
|
|
156
|
+
"target": "<target id or url>",
|
|
157
|
+
"date": "<YYYY-MM-DD>",
|
|
158
|
+
"duration_min": 42,
|
|
159
|
+
"bugs_found": 5,
|
|
160
|
+
"severity_counts": { "critical": 1, "high": 2, "medium": 2, "low": 0 },
|
|
161
|
+
"pages_explored": 14,
|
|
162
|
+
"domain": "ecommerce",
|
|
163
|
+
"started_at": "<ISO timestamp>",
|
|
164
|
+
"completed_at": "<ISO timestamp>",
|
|
165
|
+
"phases_completed": 8,
|
|
166
|
+
"total_phases": 8,
|
|
167
|
+
"coverage": { "checklist_total": 22, "checklist_verified": 15, "data_integrity_checks": 6, "data_integrity_passed": 5 },
|
|
168
|
+
"evidence": { "screenshots": 9, "console_errors_found": 2, "network_failures_found": 1 },
|
|
169
|
+
"areas_not_tested": ["settings — time ran out"],
|
|
170
|
+
"blocked_by": null
|
|
171
|
+
}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
The first eight keys are required; the rest are optional. No other top-level keys (the
|
|
175
|
+
schema is strict). Mobile puts its device block under `coverage.mobile`; backend puts the
|
|
176
|
+
verdict counts under `coverage.verdicts`.
|
|
177
|
+
|
|
178
|
+
## Index rows
|
|
179
|
+
|
|
180
|
+
Every session ends by appending one row to `output/sessions/INDEX.md` and one row per bug
|
|
181
|
+
to `output/bugs/all-bugs.md`. Both rows are written by `qualiow session finalize <session-dir>`,
|
|
182
|
+
which refuses the session — exit 1, naming the file — when an artefact is missing the
|
|
183
|
+
confidentiality header, an unredacted secret survives the scan, or `stats.json` does not
|
|
184
|
+
conform. The column order (defined once in `paths.md`) is:
|
|
185
|
+
|
|
186
|
+
```
|
|
187
|
+
| Date | Kind | Target | Bugs | Duration | Status | Report |
|
|
188
|
+
| ID | Session | Title | Severity | Status | Report |
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
A session that skips these rows is invisible to `/qa-explore-report`,
|
|
192
|
+
`/qa-explore-feedback`, `/qa-explore-cleanup` and `qualiow list sessions`.
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# Paths and Resolution Order
|
|
2
|
+
|
|
3
|
+
Every qualiow skill resolves its inputs in the same order, whichever way the skills were
|
|
4
|
+
installed (`qualiow init` copy into a project, a Claude Code plugin, or this repository
|
|
5
|
+
checked out). Resolve once in the setup phase, then **write the resolved literal prefix in
|
|
6
|
+
every command**. Never put a shell variable as the first token of a command line: Claude
|
|
7
|
+
Code's permission rules match on the first literal token, so a command that begins with a
|
|
8
|
+
variable expansion matches no rule and prompts every time. Write `qa/bin/mcli snapshot`,
|
|
9
|
+
not a variable holding that prefix.
|
|
10
|
+
|
|
11
|
+
## Target config
|
|
12
|
+
|
|
13
|
+
| Input | Resolution order (first existing wins) |
|
|
14
|
+
|---|---|
|
|
15
|
+
| `--target <name>` | `$PWD/data/targets/<name>.yml` → `${CLAUDE_PLUGIN_ROOT}/data/targets/<name>.yml` |
|
|
16
|
+
| no `--target` | `$PWD/qa/target.yml` → `<data>/targets/_default.yml` |
|
|
17
|
+
|
|
18
|
+
`qa/target.yml` is the **project-local** target: it lives next to the code under test, is
|
|
19
|
+
written by `/qa-target-setup`, and is the right place for anything with internal hostnames.
|
|
20
|
+
`data/targets/` holds shared and example targets.
|
|
21
|
+
|
|
22
|
+
## Data directory (`<data>`)
|
|
23
|
+
|
|
24
|
+
First existing of: `$PWD/data` → `${CLAUDE_PLUGIN_ROOT}/data` →
|
|
25
|
+
`${CLAUDE_SKILL_DIR}/../../data` → `${CLAUDE_SKILL_DIR}/../../../data`. The `qualiow` CLI
|
|
26
|
+
resolves the same order — an explicit `--data <dir>`, then `$PWD/data`, then
|
|
27
|
+
`$CLAUDE_PLUGIN_ROOT/data`, then the data shipped inside the package.
|
|
28
|
+
|
|
29
|
+
Read from `<data>`: `domains/<domain>.yml`, `knowledge/manifest.yml`,
|
|
30
|
+
`knowledge/releases/<version>/entries/*.yml`, `knowledge/learned-patterns.md`,
|
|
31
|
+
`templates/*.md`, `security/SECURITY-POLICY.md`.
|
|
32
|
+
|
|
33
|
+
**Writes** (learned patterns from `/qa-explore-feedback`, custom entries from
|
|
34
|
+
`/qa-knowledge-add`) always go to `$PWD/data/…`; create the directory if it is missing.
|
|
35
|
+
Never write into a plugin directory.
|
|
36
|
+
|
|
37
|
+
## Credentials
|
|
38
|
+
|
|
39
|
+
`$PWD/qa/.env` → `$PWD/.env`. Target configs name env vars; the values live only in these
|
|
40
|
+
files. Read a value when you need it, pass it to the command, and never echo it into a
|
|
41
|
+
log, a report, or a command line that will be recorded.
|
|
42
|
+
|
|
43
|
+
## Mobile driver
|
|
44
|
+
|
|
45
|
+
First existing of: `$PWD/qa/bin/mcli` → `$PWD/bin/mcli` → `mcli` on `PATH` →
|
|
46
|
+
`${CLAUDE_PLUGIN_ROOT}/bin/mcli` → `${CLAUDE_SKILL_DIR}/../../bin/mcli`. Same order for
|
|
47
|
+
`wadb`, `wk-ios` and `doctor-mobile.sh`. The mobile phase files show `qa/bin/mcli` (the
|
|
48
|
+
default after `qualiow init`); in a checkout of this repository it is `bin/mcli`.
|
|
49
|
+
Substitute the prefix you resolved.
|
|
50
|
+
|
|
51
|
+
## CLI
|
|
52
|
+
|
|
53
|
+
`qualiow` is the deterministic half of the pack: knowledge digests, session finalization,
|
|
54
|
+
listings, archival. First working of:
|
|
55
|
+
|
|
56
|
+
1. `qualiow` on `PATH` — a marketplace plugin install (its `bin/` is on `PATH`), or a global
|
|
57
|
+
npm install
|
|
58
|
+
2. `npx -y -p qualiow-exploratory-testing qualiow` — a project that depends on the npm package
|
|
59
|
+
3. `${CLAUDE_PLUGIN_ROOT}/bin/qualiow` — a plugin loaded with `--plugin-dir`
|
|
60
|
+
4. `bin/qualiow` — this repository checked out
|
|
61
|
+
|
|
62
|
+
The npm package is **`qualiow-exploratory-testing`** and the binary it installs is `qualiow`.
|
|
63
|
+
Never write bare `npx qualiow`: that spelling resolves a package that does not exist.
|
|
64
|
+
|
|
65
|
+
Same rule as the mobile driver — resolve once in setup, then write the resolved literal
|
|
66
|
+
prefix as the first token of every command. The phase files show `qualiow <command>` for
|
|
67
|
+
brevity; substitute the prefix you resolved.
|
|
68
|
+
|
|
69
|
+
## Output
|
|
70
|
+
|
|
71
|
+
Always `$PWD/output/`. Never write session output anywhere else.
|
|
72
|
+
|
|
73
|
+
### Session directory
|
|
74
|
+
|
|
75
|
+
`output/sessions/<YYYY-MM-DD-HHmm>-<kind>-<slug>/`
|
|
76
|
+
|
|
77
|
+
- `kind` ∈ `explore` | `quick` | `mobile` | `backend`
|
|
78
|
+
- `slug` = target id without a leading `_`; mobile: `<target>-<ios|android>`; backend: the
|
|
79
|
+
ticket id or the target id; an ad-hoc URL: the hostname with dots replaced by dashes.
|
|
80
|
+
Lowercase `[a-z0-9-]`, at most 40 characters.
|
|
81
|
+
- Date first, so alphabetical order is chronological and "latest" is unambiguous.
|
|
82
|
+
|
|
83
|
+
`--session <dir>`: when `qualiow explore` (pre-flight) already created the directory, reuse
|
|
84
|
+
it instead of creating a second one.
|
|
85
|
+
|
|
86
|
+
### playwright-cli session id
|
|
87
|
+
|
|
88
|
+
`-s=<kind>-<HHmm>-<slug>` (for example `-s=explore-1420-parabank`). Every `playwright-cli`
|
|
89
|
+
command in the session carries it; the phase files omit the prefix for brevity. The session
|
|
90
|
+
ends with `playwright-cli -s=<sid> close` and then `playwright-cli -s=<sid> delete-data`.
|
|
91
|
+
|
|
92
|
+
### Index rows
|
|
93
|
+
|
|
94
|
+
`qualiow session finalize <session-dir>` writes both rows (and creates either index file if
|
|
95
|
+
it is missing). The formats below are what it appends — write them by hand only when the CLI
|
|
96
|
+
is unavailable.
|
|
97
|
+
|
|
98
|
+
Append to `output/sessions/INDEX.md` (columns `| Date | Kind | Target | Bugs | Duration | Status | Report |`):
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
| <YYYY-MM-DD> | <kind> | <target> | <bugs> | <N> min | complete | <session-dir>/session-report.md |
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Append one row per bug to `output/bugs/all-bugs.md` (columns `| ID | Session | Title | Severity | Status | Report |`):
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
| BUG-NNN | <session-dir> | <title> | <severity> | open | <session-dir>/bugs/BUG-NNN.md |
|
|
108
|
+
```
|
|
@@ -1,43 +1,43 @@
|
|
|
1
1
|
# Playwright Test Agents Integration (opt-in)
|
|
2
2
|
|
|
3
|
-
> **This file is opt-in reference material.** Read it only when the
|
|
3
|
+
> **This file is opt-in reference material.** Read it only when the consumer project has
|
|
4
|
+
> `@playwright/test` installed and a `playwright.config.*` file. Otherwise skip it;
|
|
5
|
+
> `qa-explore` works end-to-end without it.
|
|
4
6
|
|
|
5
7
|
## Purpose
|
|
6
8
|
|
|
7
|
-
`qa-explore` is an **exploratory** skill:
|
|
9
|
+
`qa-explore` is an **exploratory** skill: blind (or context-enriched) discovery of bugs,
|
|
10
|
+
risks and missing functionality in a web application, bounded by a 45-minute session.
|
|
8
11
|
|
|
9
|
-
**Playwright Test Agents** (
|
|
10
|
-
|
|
11
|
-
|
|
12
|
+
**Playwright Test Agents** (Playwright 1.56+) are a different, complementary capability:
|
|
13
|
+
three agent definitions, **planner**, **generator** and **healer**, that author and maintain
|
|
14
|
+
executable Playwright test specs.
|
|
12
15
|
|
|
13
16
|
| Concern | `qa-explore` | Playwright Test Agents |
|
|
14
17
|
|---|---|---|
|
|
15
18
|
| Goal | Find unknown bugs and risks | Generate and maintain regression tests |
|
|
16
19
|
| Output | Bug reports, session report, coverage map | `tests/*.spec.ts` files |
|
|
17
|
-
| When to use | New feature, new app, post-incident, pre-release audit | Known
|
|
18
|
-
| Input required | Just a URL (context optional) | A plan
|
|
20
|
+
| When to use | New feature, new app, post-incident, pre-release audit | Known behaviour you want locked in with a test |
|
|
21
|
+
| Input required | Just a URL (context optional) | A plan, a spec, or a reproducible behaviour |
|
|
19
22
|
|
|
20
23
|
Use `qa-explore` to discover; use Test Agents to codify a discovery into a regression test.
|
|
21
24
|
|
|
22
25
|
## When to use this handoff
|
|
23
26
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
-
|
|
27
|
-
- The user completed a `qa-explore` session and explicitly asked "can you write a test for this bug?"
|
|
28
|
-
- The user wants to seed a new test project from a qa-explore charter.
|
|
29
|
-
|
|
30
|
-
You are in the WRONG file if:
|
|
27
|
+
- Phase 5 or 6 produced a **reproducible** bug that deserves a regression test.
|
|
28
|
+
- The user completed a session and asked "can you write a test for this bug?"
|
|
29
|
+
- The user wants to seed a new test project from a session charter.
|
|
31
30
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
- The bug is not reproducible. Test Agents need deterministic input; flaky observations go in the bug report only.
|
|
31
|
+
Not mid-session (finish `qa-explore` first), not without `@playwright/test`, and not for a
|
|
32
|
+
bug that is not reproducible (flaky observations belong in the bug report only).
|
|
35
33
|
|
|
36
34
|
## Prerequisites
|
|
37
35
|
|
|
38
|
-
1. `@playwright/test ^1.59.1` installed in the consumer project (
|
|
39
|
-
|
|
40
|
-
|
|
36
|
+
1. `@playwright/test ^1.59.1` installed in the consumer project (an optional peer
|
|
37
|
+
dependency of this package).
|
|
38
|
+
2. A `playwright.config.ts` or `playwright.config.js` in the consumer project. If absent,
|
|
39
|
+
run `npm init playwright@latest` first.
|
|
40
|
+
3. A completed session with a bug report that has exact reproduction steps.
|
|
41
41
|
|
|
42
42
|
## Bootstrap
|
|
43
43
|
|
|
@@ -47,80 +47,57 @@ From the consumer project root:
|
|
|
47
47
|
npx playwright init-agents --loop=claude
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
- Agent definition files for the selected loop (`claude`, `vscode`, or `opencode` are supported)
|
|
53
|
-
- A minimal `tests/agents/` scaffolding directory
|
|
54
|
-
- Updates to `playwright.config.ts` to wire agent reporters
|
|
55
|
-
|
|
56
|
-
The `--loop=claude` flag optimizes the agent definitions for Claude Code. `--loop=vscode` optimizes them for VS Code's built-in Copilot integration. Pick the one that matches the consumer's environment.
|
|
57
|
-
|
|
58
|
-
This command is idempotent — re-running it updates definitions without deleting committed tests.
|
|
59
|
-
|
|
60
|
-
## The three agents
|
|
61
|
-
|
|
62
|
-
### Planner
|
|
63
|
-
|
|
64
|
-
**Role:** Given a description of desired behavior (from a charter, a PRD, or a bug repro), produce an ordered plan of tests to write.
|
|
65
|
-
|
|
66
|
-
**Hand-off from qa-explore:** After a session, if the charter identified P0 journeys that have NO automated coverage, feed the charter and coverage map to the planner. The output is a list of test files the team should author.
|
|
50
|
+
Supported loops: `claude`, `codex`, `copilot`, `opencode`, `vscode`, `vscode-legacy`.
|
|
51
|
+
With `--loop=claude` it writes:
|
|
67
52
|
|
|
68
|
-
|
|
53
|
+
- `.claude/agents/` with one Markdown definition each for the planner, generator and healer
|
|
54
|
+
- `.mcp.json` registering the `playwright-test` server (`npx playwright run-test-mcp-server`)
|
|
55
|
+
- `specs/` with a `README.md`, and a default seed spec file if the project has none
|
|
56
|
+
(`--prompts` additionally writes `.claude/prompts/`)
|
|
69
57
|
|
|
70
|
-
|
|
58
|
+
It does not create a `tests/agents/` directory and does not edit `playwright.config.*`.
|
|
59
|
+
The definitions are generated files: re-run the command after upgrading Playwright so the
|
|
60
|
+
agents pick up new tools and instructions.
|
|
71
61
|
|
|
72
|
-
|
|
62
|
+
## The three agents (upstream definitions)
|
|
73
63
|
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
**
|
|
81
|
-
|
|
82
|
-
**Hand-off from qa-explore:** Not a direct handoff. Used by the consumer's team later when a generated spec starts failing due to non-bug changes.
|
|
64
|
+
- **Planner** explores the app and produces a Markdown test plan. Hand-off from
|
|
65
|
+
`qa-explore`: feed it the charter and the coverage map for P0 journeys that have no
|
|
66
|
+
automated coverage.
|
|
67
|
+
- **Generator** transforms the Markdown plan into Playwright Test files. Hand-off: pass a
|
|
68
|
+
reproducible bug's steps and expected-vs-actual; it writes `tests/repro-BUG-NNN.spec.ts`,
|
|
69
|
+
which fails until the fix lands.
|
|
70
|
+
- **Healer** executes the test suite and automatically repairs failing tests. No direct
|
|
71
|
+
hand-off; the team uses it later when a generated spec breaks for non-bug reasons.
|
|
83
72
|
|
|
84
73
|
## Bug → regression test workflow
|
|
85
74
|
|
|
86
|
-
1. **In
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
## Complementary tooling note
|
|
94
|
-
|
|
95
|
-
The official **Playwright MCP server** (see `github.com/microsoft/playwright-mcp`) is an alternative runtime harness for LLM-driven browser automation via accessibility trees. It overlaps with `playwright-cli` but targets long-lived agent sessions with persistent browser state. It is NOT required for Test Agents or `qa-explore` — mentioned here only so the consumer knows it exists.
|
|
75
|
+
1. **In phase 5 or 6**, confirm the bug is reproducible: trigger it twice with the same
|
|
76
|
+
steps and identical results.
|
|
77
|
+
2. **In phase 7**, write `bugs/BUG-NNN.md` per `output-contract.md`.
|
|
78
|
+
3. **After the session**, invoke the generator agent with the bug report as input.
|
|
79
|
+
4. **Run the new spec once**: `npx playwright test tests/repro-BUG-NNN.spec.ts`. It should
|
|
80
|
+
fail while the bug exists. Commit it with the bug report.
|
|
96
81
|
|
|
97
82
|
## Limitations
|
|
98
83
|
|
|
99
|
-
-
|
|
100
|
-
|
|
101
|
-
-
|
|
102
|
-
-
|
|
84
|
+
- Opt-in only. This file, the `Bash(npx playwright:*)` permission and the peer dependency
|
|
85
|
+
are all optional; sessions that never touch them are unaffected.
|
|
86
|
+
- Test Agents turn known behaviour into specs. They do not replace exploratory discovery.
|
|
87
|
+
- The generator needs deterministic reproduction steps.
|
|
103
88
|
|
|
104
89
|
## Feature cross-reference
|
|
105
90
|
|
|
106
|
-
Playwright
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
| `npx playwright init-agents` | 1.56 |
|
|
112
|
-
| `page.consoleMessages()` | 1.
|
|
113
|
-
| `page.
|
|
114
|
-
| `page.
|
|
115
|
-
|
|
|
116
|
-
|
|
|
117
|
-
| `
|
|
118
|
-
|
|
|
119
|
-
| Trace Viewer themes | 1.58 | `phases/07-reporting.md` |
|
|
120
|
-
| `page.ariaSnapshot()` | 1.59 | `phases/06-edge-cases.md` |
|
|
121
|
-
| `page.pickLocator()` | 1.59 | `phases/05-features.md`, `qa-explore-quick/SKILL.md` |
|
|
122
|
-
| `locator.normalize()` | 1.59 | `phases/05-features.md`, `qa-explore-quick/SKILL.md` |
|
|
123
|
-
| `npx playwright trace` CLI | 1.59 | `phases/07-reporting.md` |
|
|
124
|
-
| `page.screencast()` | 1.59 | `phases/07-reporting.md` |
|
|
125
|
-
| `browser.bind()` | 1.59 | (reference only, not in phase guidance) |
|
|
126
|
-
| `--debug=cli` flag | 1.59 | (reference only, not in phase guidance) |
|
|
91
|
+
Playwright APIs the phases mention, with the version that introduced them, and how each is
|
|
92
|
+
reached from a `playwright-cli` session:
|
|
93
|
+
|
|
94
|
+
| Feature | Version | Used in | How |
|
|
95
|
+
|---|---|---|---|
|
|
96
|
+
| `npx playwright init-agents` | 1.56 | `${CLAUDE_SKILL_DIR}/phases/00-setup.md`, this file | shell |
|
|
97
|
+
| `page.consoleMessages()`, `page.pageErrors()`, `page.requests()` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/03-discovery.md` | `playwright-cli run-code "async page => …"` |
|
|
98
|
+
| `page.accessibility` **removed** | 1.57 | `${CLAUDE_SKILL_DIR}/phases/06-edge-cases.md` | use `snapshot` (aria tree) or `run-code` + `page.ariaSnapshot()` |
|
|
99
|
+
| `locator.ariaSnapshot()` (1.49), `page.ariaSnapshot()` (1.59) | 1.49 / 1.59 | `${CLAUDE_SKILL_DIR}/phases/06-edge-cases.md` | `run-code` |
|
|
100
|
+
| `page.pickLocator()` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/05-features.md` | human-only (opens a picker in a headed browser); agents use `generate-locator <ref>` |
|
|
101
|
+
| `page.screencast` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/07-reporting.md` | `video-start` / `video-chapter`, or a `run-code --filename` hero script |
|
|
102
|
+
| `npx playwright trace open <trace>` | 1.59 | `${CLAUDE_SKILL_DIR}/phases/07-reporting.md` | shell |
|
|
103
|
+
| `--debug=cli`, `browser.bind()` | 1.59 | reference only | not used by the session |
|