qualiow-exploratory-testing 1.2.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +22 -0
- package/.env.example +27 -0
- package/CHANGELOG.md +194 -0
- package/README.md +428 -179
- package/agents/qa-gather-agent.md +24 -0
- package/{scripts → bin}/doctor-mobile.sh +57 -18
- package/bin/mcli +15 -9
- package/bin/mobile-cli.mjs +574 -286
- package/{scripts → bin}/setup-mobile.sh +56 -19
- package/bin/wadb +10 -4
- package/bin/wk-ios +29 -17
- package/bin/wkeval.mjs +7 -1
- package/data/domains/_default.yml +1 -1
- package/data/domains/identity.yml +138 -0
- package/data/knowledge/changelog.yml +104 -0
- package/data/knowledge/learned-patterns.md +172 -86
- package/data/knowledge/manifest.yml +481 -70
- package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
- package/data/knowledge/releases/v0.1.0/release.yml +8 -1
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
- package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
- package/data/knowledge/releases/v0.6.0/release.yml +30 -0
- package/data/security/SECURITY-POLICY.md +273 -0
- package/data/targets/_default.yml +1 -1
- package/data/targets/_example-api-only.yml +80 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +2 -2
- package/data/targets/_example-native-mobile.yml +2 -2
- package/data/targets/_example-sim-android-chrome.yml +1 -1
- package/data/targets/_example-sim-ios-safari.yml +1 -1
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/bug-report.md +29 -151
- package/data/templates/coverage-map.md +23 -6
- package/data/templates/expected-behaviour.md +94 -0
- package/data/templates/session-charter.md +3 -0
- package/data/templates/session-report.md +57 -80
- package/dist/cli/index.js +1500 -739
- package/dist/index.d.ts +673 -75
- package/dist/index.js +1664 -270
- package/docs/BACKEND-VERIFICATION.md +307 -0
- package/docs/GETTING-STARTED.md +188 -128
- package/docs/MOBILE-SETUP.md +149 -18
- package/package.json +32 -16
- package/skills/qa-explore/SKILL.md +31 -25
- package/skills/qa-explore/phases/00-setup.md +67 -41
- package/skills/qa-explore/phases/01-auth.md +32 -15
- package/skills/qa-explore/phases/02-charter.md +16 -5
- package/skills/qa-explore/phases/03-discovery.md +50 -26
- package/skills/qa-explore/phases/04-journeys.md +2 -2
- package/skills/qa-explore/phases/05-features.md +39 -24
- package/skills/qa-explore/phases/06-edge-cases.md +24 -13
- package/skills/qa-explore/phases/07-reporting.md +79 -140
- package/skills/qa-explore/references/context-integration.md +1 -1
- package/skills/qa-explore/references/output-contract.md +187 -0
- package/skills/qa-explore/references/paths.md +84 -0
- package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
- package/skills/qa-explore/references/security-rules.md +66 -25
- package/skills/qa-explore/references/severity-guide.md +31 -4
- package/skills/qa-explore-cleanup/SKILL.md +17 -6
- package/skills/qa-explore-feedback/SKILL.md +18 -5
- package/skills/qa-explore-mobile/SKILL.md +86 -64
- package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
- package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
- package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
- package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
- package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
- package/skills/qa-explore-mobile/phases/05-features.md +9 -9
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
- package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
- package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
- package/skills/qa-explore-quick/SKILL.md +60 -25
- package/skills/qa-explore-report/SKILL.md +40 -34
- package/skills/qa-gather/SKILL.md +17 -2
- package/skills/qa-knowledge-add/SKILL.md +20 -9
- package/skills/qa-knowledge-list/SKILL.md +12 -3
- package/skills/qa-target-setup/SKILL.md +63 -41
- package/skills/qa-verify-backend/SKILL.md +172 -0
- package/skills/qa-verify-backend/phases/00-setup.md +142 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
- package/skills/qa-verify-backend/references/api-probes.md +168 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +95 -0
- package/dist/cli/index.d.ts +0 -1
- package/dist/cli/index.js.map +0 -1
- package/dist/index.js.map +0 -1
package/docs/GETTING-STARTED.md
CHANGED
|
@@ -2,123 +2,179 @@
|
|
|
2
2
|
|
|
3
3
|
Step-by-step guide to running your first AI-powered exploratory testing session.
|
|
4
4
|
|
|
5
|
-
## 1.
|
|
5
|
+
## 1. Install
|
|
6
6
|
|
|
7
7
|
### System requirements
|
|
8
8
|
|
|
9
|
-
- Node.js >=
|
|
10
|
-
-
|
|
11
|
-
-
|
|
9
|
+
- **Node.js >= 22.4** (`node --version`)
|
|
10
|
+
- **Claude Code**, installed and configured — the skills run as slash commands inside it
|
|
11
|
+
- `playwright-cli` arrives as a dependency of this package. There is nothing else to install
|
|
12
|
+
for the web path.
|
|
12
13
|
|
|
13
14
|
### Install the package
|
|
14
15
|
|
|
15
16
|
```bash
|
|
16
|
-
npm install -g
|
|
17
|
+
npm install -g qualiow-exploratory-testing
|
|
18
|
+
qualiow --version # prints the installed package version, e.g. 2.0.0
|
|
17
19
|
```
|
|
18
20
|
|
|
19
|
-
###
|
|
20
|
-
|
|
21
|
-
Playwright CLI is the browser control layer. Install it as a project dependency or globally.
|
|
21
|
+
### Initialise a project
|
|
22
22
|
|
|
23
23
|
```bash
|
|
24
|
-
|
|
24
|
+
cd my-project
|
|
25
|
+
qualiow init
|
|
25
26
|
```
|
|
26
27
|
|
|
27
|
-
|
|
28
|
+
That writes:
|
|
29
|
+
|
|
30
|
+
| What | Where |
|
|
31
|
+
|------|-------|
|
|
32
|
+
| the 11 skills | `.claude/skills/qa-*/` |
|
|
33
|
+
| the `qa-gather-agent` sub-agent | `.claude/agents/qa-gather-agent.md` |
|
|
34
|
+
| knowledge base, domain profiles, templates, security policy | `data/knowledge/`, `data/domains/`, `data/templates/`, `data/security/` |
|
|
35
|
+
| the `_default` target (add `--include-examples` for the `_example-*` templates and `testers-ai.yml`) | `data/targets/` |
|
|
36
|
+
| the mobile driver plus `setup-mobile.sh` and `doctor-mobile.sh`, executable | `qa/bin/` |
|
|
37
|
+
| the credential template | `qa/.env.example` |
|
|
38
|
+
| empty session/bug/context trees and the auth directory | `output/sessions/`, `output/bugs/`, `output/context/`, `.auth/` |
|
|
39
|
+
| a `# Qualiow` block appended to `.gitignore` | `.gitignore` |
|
|
40
|
+
|
|
41
|
+
Re-running `qualiow init` changes nothing unless you pass `--force`. Use `--dry-run` to see
|
|
42
|
+
the plan first.
|
|
28
43
|
|
|
29
44
|
```bash
|
|
30
|
-
|
|
45
|
+
cp qa/.env.example qa/.env # credentials go here, never in a YAML file
|
|
46
|
+
npx playwright-cli install --skills # optional: the official Playwright skill alongside
|
|
31
47
|
```
|
|
32
48
|
|
|
33
|
-
|
|
49
|
+
Alternative install paths — the Claude Code plugin (`claude --plugin-dir <repo>`, skills
|
|
50
|
+
appear as `/qualiow:qa-explore`) and a plain git checkout — are described in the project
|
|
51
|
+
README.
|
|
34
52
|
|
|
35
|
-
|
|
53
|
+
## 2. Run your first session
|
|
36
54
|
|
|
37
|
-
|
|
55
|
+
### Option A: explore a public site
|
|
38
56
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
| SauceDemo | https://www.saucedemo.com | ecommerce |
|
|
42
|
-
| ParaBank | https://parabank.parasoft.com/parabank/index.htm | fintech |
|
|
43
|
-
| DemoQA | https://demoqa.com | general |
|
|
44
|
-
|
|
45
|
-
Start a session inside Claude Code:
|
|
57
|
+
Point `/qa-explore` at any public site. A good first target is
|
|
58
|
+
<https://testers.ai/testing/>, which ships as the `testers-ai` target config.
|
|
46
59
|
|
|
47
60
|
```bash
|
|
48
|
-
/qa-explore https://
|
|
61
|
+
/qa-explore https://testers.ai/testing/
|
|
49
62
|
```
|
|
50
63
|
|
|
51
64
|
Claude will:
|
|
52
|
-
1. Launch a browser via Playwright CLI
|
|
53
|
-
2. Read the knowledge base for applicable heuristics
|
|
54
|
-
3. Explore the site across 3-4 phases (discovery, functional, edge cases, synthesis)
|
|
55
|
-
4. File individual bug reports in `output/sessions/<session-id>/bugs/`
|
|
56
|
-
5. Write a session report at `output/sessions/<session-id>/session-report.md`
|
|
57
|
-
|
|
58
|
-
### Option B: Quick check on a specific page
|
|
59
65
|
|
|
60
|
-
|
|
66
|
+
1. Resolve the target config and credentials, and create the session directory
|
|
67
|
+
2. Authenticate if the target requires it
|
|
68
|
+
3. Write a charter — what is being tested, why, and what "done" looks like
|
|
69
|
+
4. Read the knowledge base for the heuristics that apply to this domain
|
|
70
|
+
5. Explore across eight phases, saving findings to disk between each
|
|
71
|
+
6. File one bug report per finding in `bugs/`
|
|
72
|
+
7. Write `session-report.md` and update the session index
|
|
73
|
+
|
|
74
|
+
The eight phases and their share of the 45-minute cap:
|
|
75
|
+
|
|
76
|
+
| # | Phase | Minutes | What happens |
|
|
77
|
+
|---|-------|---------|--------------|
|
|
78
|
+
| 0 | Setup | 2 | Resolve config, create session dir, open the browser session |
|
|
79
|
+
| 1 | Auth | 2 | Log in (storage state or adaptive login) |
|
|
80
|
+
| 2 | Charter | 5 | Understand the business, risk-rank features P0–P3 |
|
|
81
|
+
| 3 | Discovery | 6 | Map the app: pages, forms, roles, console and network health |
|
|
82
|
+
| 4 | Journeys | 10 | End-to-end user journeys with data-integrity checks |
|
|
83
|
+
| 5 | Features | 10 | Deep testing of the highest-risk features |
|
|
84
|
+
| 6 | Edge cases | 6 | Boundaries, negative input, security, "what is missing?" |
|
|
85
|
+
| 7 | Reporting | 4 | Reflection, coverage map, session report, cleanup |
|
|
86
|
+
|
|
87
|
+
### Option B: quick check on one page
|
|
61
88
|
|
|
62
89
|
```bash
|
|
63
90
|
/qa-explore-quick https://app.example.com/checkout
|
|
64
91
|
```
|
|
65
92
|
|
|
66
|
-
|
|
93
|
+
A 15-minute focused session — no full site mapping. It produces the same artefacts as a full
|
|
94
|
+
session, so reports, feedback and cleanup all work on it.
|
|
95
|
+
|
|
96
|
+
### Option C: give it the requirements first
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
/qa-gather
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Point it at a ticket, a PR, a design doc, a URL or pasted text. It writes
|
|
103
|
+
`output/context/<TICKET>-context.md`, which you then pass to a session:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
/qa-explore --target my-app --context output/context/TICKET-123-context.md
|
|
107
|
+
```
|
|
67
108
|
|
|
68
|
-
## 3.
|
|
109
|
+
## 3. Understand the output
|
|
69
110
|
|
|
70
|
-
|
|
111
|
+
Every session — web, quick, mobile or backend — lands in
|
|
112
|
+
`output/sessions/<YYYY-MM-DD-HHmm>-<kind>-<slug>/`, where `kind` is `explore`, `quick`,
|
|
113
|
+
`mobile` or `backend`:
|
|
71
114
|
|
|
72
115
|
```
|
|
73
|
-
output/sessions
|
|
74
|
-
charter.md # what
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
phase-3-
|
|
78
|
-
|
|
116
|
+
output/sessions/2026-09-08-1813-explore-parabank/
|
|
117
|
+
charter.md # what is being tested and why
|
|
118
|
+
session-log.md # real-time findings, with the WHY not just the WHAT
|
|
119
|
+
progress.json
|
|
120
|
+
phase-3-discovery.md # phase artefacts — the file number is the phase number
|
|
121
|
+
phase-4-journeys.md
|
|
122
|
+
phase-5-features.md
|
|
123
|
+
phase-6-edge-cases.md
|
|
79
124
|
bugs/
|
|
80
|
-
BUG-001.md #
|
|
125
|
+
BUG-001.md # one bug = one report
|
|
81
126
|
BUG-002.md
|
|
82
|
-
...
|
|
83
127
|
screenshots/
|
|
84
|
-
|
|
85
|
-
|
|
128
|
+
BUG-001.png
|
|
129
|
+
videos/
|
|
130
|
+
session-report.md # the deliverable
|
|
131
|
+
stats.json # machine-readable session metrics
|
|
86
132
|
```
|
|
87
133
|
|
|
134
|
+
Plus, at the top level:
|
|
135
|
+
|
|
136
|
+
- `output/sessions/INDEX.md` — one row per session
|
|
137
|
+
(`| Date | Kind | Target | Bugs | Duration | Status | Report |`)
|
|
138
|
+
- `output/bugs/all-bugs.md` — every bug across every session
|
|
139
|
+
|
|
88
140
|
### Bug report structure
|
|
89
141
|
|
|
90
|
-
Each
|
|
142
|
+
Each report opens with the confidentiality blockquote, then:
|
|
91
143
|
|
|
92
|
-
- **Title**
|
|
93
|
-
- **Severity**
|
|
94
|
-
|
|
95
|
-
- **Expected
|
|
96
|
-
- **
|
|
97
|
-
- **
|
|
144
|
+
- **Title** — `# BUG-NNN: [Component] fails [Condition] causing [Impact]`
|
|
145
|
+
- **Severity** — Critical, High, Medium or Low (conservative by design) — plus Priority,
|
|
146
|
+
Component, URL, Environment and reproduction rate
|
|
147
|
+
- **Summary**, **Expected Behavior**, **Actual Behavior**
|
|
148
|
+
- **Steps to Reproduce** — numbered, specific enough for someone else to follow
|
|
149
|
+
- **Business Impact** — revenue, trust, regulatory, data and scale. Mandatory
|
|
150
|
+
- **Evidence** — screenshot, video, logs, console errors, network failures
|
|
151
|
+
- **Recommended Fix Priority**
|
|
98
152
|
|
|
99
153
|
### Session report structure
|
|
100
154
|
|
|
101
|
-
|
|
155
|
+
- Session metadata and executive summary
|
|
156
|
+
- Summary stats and the bug table
|
|
157
|
+
- **Coverage map** (`| Area | Risk | Status | Bugs | Notes |`) — including what was *not* tested
|
|
158
|
+
- Observations that are not bugs but are worth discussing
|
|
159
|
+
- Recommendations, reflection, session stats
|
|
102
160
|
|
|
103
|
-
|
|
104
|
-
- Bug table with severity distribution
|
|
105
|
-
- Coverage map showing which areas were tested
|
|
106
|
-
- Observations (not bugs, but worth discussing)
|
|
107
|
-
- Recommendations for follow-up testing
|
|
161
|
+
## 4. Configure a target
|
|
108
162
|
|
|
109
|
-
|
|
163
|
+
For repeated testing of the same application, save a target config.
|
|
110
164
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
### Generate a target config interactively
|
|
165
|
+
### Interactively
|
|
114
166
|
|
|
115
167
|
```bash
|
|
116
168
|
/qa-target-setup
|
|
117
169
|
```
|
|
118
170
|
|
|
119
|
-
|
|
171
|
+
It navigates to the URL, detects the auth requirement, walks you through the login flow, and
|
|
172
|
+
writes `qa/target.yml` — the project-local target. Pass `--shared` to write
|
|
173
|
+
`data/targets/<id>.yml` instead, for a config the whole team uses.
|
|
174
|
+
|
|
175
|
+
### Or by hand
|
|
120
176
|
|
|
121
|
-
|
|
177
|
+
`qa/target.yml`:
|
|
122
178
|
|
|
123
179
|
```yaml
|
|
124
180
|
id: my-app
|
|
@@ -147,93 +203,84 @@ scope:
|
|
|
147
203
|
- /api/*
|
|
148
204
|
```
|
|
149
205
|
|
|
150
|
-
|
|
206
|
+
Validate it before you use it:
|
|
151
207
|
|
|
152
208
|
```bash
|
|
153
|
-
|
|
209
|
+
qualiow validate --all
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
### Resolution order
|
|
213
|
+
|
|
214
|
+
```
|
|
215
|
+
--target <name> → qa/target.yml → data/targets/_default.yml
|
|
154
216
|
```
|
|
155
217
|
|
|
218
|
+
Domains come from `data/domains/*.yml` (`_default`, `ecommerce`, `fintech`, `identity`,
|
|
219
|
+
`marketing`, `saas`), and drive the risk ranking, completeness checklist and data-integrity
|
|
220
|
+
checks a session applies.
|
|
221
|
+
|
|
156
222
|
### Authentication strategies
|
|
157
223
|
|
|
158
224
|
| Strategy | When to use | Setup |
|
|
159
225
|
|----------|-------------|-------|
|
|
160
226
|
| `none` | Public sites, no login required | Default |
|
|
161
|
-
| `storage_state` |
|
|
162
|
-
| `credentials` | Username/password login |
|
|
163
|
-
| `token` | API token or bearer auth |
|
|
227
|
+
| `storage_state` | Cookie/session auth | Log in once by hand, then `playwright-cli state-save .auth/<id>.json` |
|
|
228
|
+
| `credentials` | Username/password login | Values in `qa/.env`, referenced by variable name |
|
|
229
|
+
| `token` | API token or bearer auth | Value in `qa/.env`, referenced by variable name |
|
|
164
230
|
|
|
165
|
-
|
|
231
|
+
### Credentials
|
|
166
232
|
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
MY_APP_USERNAME=testuser
|
|
170
|
-
MY_APP_PASSWORD=testpass123
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
## 5. How to Add Knowledge
|
|
174
|
-
|
|
175
|
-
The knowledge base drives Claude's testing strategy. Add domain-specific heuristics, techniques, or checklists.
|
|
176
|
-
|
|
177
|
-
### Add knowledge interactively
|
|
233
|
+
Secrets live in `qa/.env` (gitignored), falling back to `.env`. YAML files reference them **by
|
|
234
|
+
variable name only** — never by value. The names the shipped configs use:
|
|
178
235
|
|
|
179
236
|
```bash
|
|
180
|
-
|
|
237
|
+
# qa/.env
|
|
238
|
+
QA_USER=testuser
|
|
239
|
+
QA_PASS=...
|
|
240
|
+
QA_TOKEN=...
|
|
241
|
+
|
|
242
|
+
# /qa-verify-backend
|
|
243
|
+
QA_AWS_PROFILE=... # read-only profile. Never point this at production
|
|
244
|
+
QA_AWS_REGION=eu-west-1
|
|
245
|
+
QA_API_TOKEN=... # only for a target declaring api.auth: bearer-env
|
|
246
|
+
EXAMPLE_API_KEY=... # a no-UI service names its own variable; see _example-api-only.yml
|
|
181
247
|
```
|
|
182
248
|
|
|
183
|
-
|
|
249
|
+
## 5. Add knowledge
|
|
250
|
+
|
|
251
|
+
The knowledge base drives Claude's testing strategy: 29 entries across releases v0.1.0–v0.6.0,
|
|
252
|
+
indexed by `data/knowledge/manifest.yml`.
|
|
184
253
|
|
|
185
254
|
```bash
|
|
186
|
-
/qa-knowledge-
|
|
255
|
+
/qa-knowledge-add # add a heuristic, technique, checklist or reference
|
|
256
|
+
/qa-knowledge-list # browse what is there, by type, domain or tag
|
|
187
257
|
```
|
|
188
258
|
|
|
189
|
-
### Knowledge types
|
|
190
|
-
|
|
191
259
|
| Type | Purpose | Example |
|
|
192
260
|
|------|---------|---------|
|
|
193
261
|
| `heuristic` | Mental model or framework for exploration | SFDIPOT, FEW HICCUPPS |
|
|
194
|
-
| `technique` |
|
|
262
|
+
| `technique` | A specific testing method | boundary testing, authenticated API probing |
|
|
195
263
|
| `checklist` | Verification list for a specific area | WCAG accessibility checks |
|
|
196
264
|
| `reference` | Background material | exploratory testing type definitions |
|
|
197
265
|
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
## 6. How to Use CLI Commands
|
|
266
|
+
Entries are YAML files in `data/knowledge/releases/<version>/entries/`. After editing by hand,
|
|
267
|
+
run `qualiow kb sync` to regenerate the manifest and `qualiow kb check` to verify it.
|
|
201
268
|
|
|
202
|
-
|
|
269
|
+
## 6. CLI commands
|
|
203
270
|
|
|
204
271
|
```bash
|
|
205
|
-
qualiow
|
|
272
|
+
qualiow init --dry-run # preview what init would write
|
|
273
|
+
qualiow validate --all # targets + qa/target.yml + domains + knowledge base
|
|
274
|
+
qualiow list sessions # also: knowledge | targets | domains
|
|
275
|
+
qualiow report -s latest -f html -o report.html
|
|
276
|
+
qualiow kb check
|
|
206
277
|
```
|
|
207
278
|
|
|
208
|
-
|
|
279
|
+
`qualiow explore [url]` is **pre-flight only**: it validates the inputs, creates the session
|
|
280
|
+
directory, and prints the `/qa-explore … --session <dir>` command to run inside Claude Code.
|
|
281
|
+
It does not drive a browser itself.
|
|
209
282
|
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
```bash
|
|
213
|
-
# list all sessions
|
|
214
|
-
qualiow list sessions
|
|
215
|
-
|
|
216
|
-
# list knowledge entries
|
|
217
|
-
qualiow list knowledge
|
|
218
|
-
|
|
219
|
-
# list configured targets
|
|
220
|
-
qualiow list targets
|
|
221
|
-
|
|
222
|
-
# list domain profiles
|
|
223
|
-
qualiow list domains
|
|
224
|
-
```
|
|
225
|
-
|
|
226
|
-
### Generate reports from an existing session
|
|
227
|
-
|
|
228
|
-
```bash
|
|
229
|
-
qualiow report
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
Or use the skill for more control:
|
|
233
|
-
|
|
234
|
-
```bash
|
|
235
|
-
/qa-explore-report output/sessions/2026-03-28-parabank
|
|
236
|
-
```
|
|
283
|
+
There is no `qualiow gather` — it was removed in 2.0.0. Use the `/qa-gather` skill.
|
|
237
284
|
|
|
238
285
|
### Export session data programmatically
|
|
239
286
|
|
|
@@ -242,24 +289,37 @@ import {
|
|
|
242
289
|
generateHtmlReport,
|
|
243
290
|
generateJsonReport,
|
|
244
291
|
generateJiraExport,
|
|
245
|
-
} from '
|
|
292
|
+
} from 'qualiow-exploratory-testing';
|
|
293
|
+
|
|
294
|
+
const session = 'output/sessions/2026-09-08-1813-explore-parabank';
|
|
246
295
|
|
|
247
296
|
// HTML report (standalone, dark-mode, print-friendly)
|
|
248
|
-
const html = await generateHtmlReport(
|
|
297
|
+
const html = await generateHtmlReport(session);
|
|
249
298
|
|
|
250
299
|
// JSON report (structured data for dashboards and CI)
|
|
251
|
-
const json = await generateJsonReport(
|
|
300
|
+
const json = await generateJsonReport(session);
|
|
252
301
|
|
|
253
302
|
// Jira CSV (bulk import into Jira)
|
|
254
|
-
const csv = await generateJiraExport(
|
|
303
|
+
const csv = await generateJiraExport(session);
|
|
255
304
|
```
|
|
256
305
|
|
|
257
|
-
|
|
306
|
+
Every export carries the confidentiality header and runs through the same redaction list the
|
|
307
|
+
skills apply.
|
|
308
|
+
|
|
309
|
+
## 7. Beyond the desktop browser
|
|
310
|
+
|
|
311
|
+
- **Mobile** — `/qa-explore-mobile` runs a session on an iOS Simulator or Android Emulator,
|
|
312
|
+
against a native app or a web app in the real device browser. The machine needs the mobile
|
|
313
|
+
toolchain first: run `qa/bin/setup-mobile.sh`, then `qa/bin/doctor-mobile.sh` until it says
|
|
314
|
+
READY. Full guide: **MOBILE-SETUP.md**.
|
|
315
|
+
- **Backend, API and infrastructure** — `/qa-verify-backend` verifies acceptance criteria that
|
|
316
|
+
never reach a screen and produces an AC traceability matrix. Full guide:
|
|
317
|
+
**BACKEND-VERIFICATION.md**.
|
|
258
318
|
|
|
259
|
-
|
|
319
|
+
## What to do next
|
|
260
320
|
|
|
261
|
-
1. Review the session report and bug reports
|
|
262
|
-
2.
|
|
263
|
-
3. Configure a target for your own application
|
|
264
|
-
4. Add domain-specific knowledge
|
|
321
|
+
1. Review the session report and the bug reports
|
|
322
|
+
2. Run `/qa-explore-feedback` to mark false positives or note bugs the session missed
|
|
323
|
+
3. Configure a target for your own application with `/qa-target-setup`
|
|
324
|
+
4. Add domain-specific knowledge so the next session starts smarter
|
|
265
325
|
5. Run `/qa-explore-cleanup` to archive completed sessions
|
package/docs/MOBILE-SETUP.md
CHANGED
|
@@ -6,32 +6,35 @@ The mobile skill drives **real** simulator/emulator browsers and native apps via
|
|
|
6
6
|
and/or Android toolchains being installed on the machine. This guide gets a clean Mac to
|
|
7
7
|
the point where a session can start.
|
|
8
8
|
|
|
9
|
+
Paths below are written as `bin/…` (a git checkout or a plugin install). In a project you
|
|
10
|
+
ran `qualiow init` in, the same files are at `qa/bin/…`; a global npm install also puts
|
|
11
|
+
`mcli`, `wadb`, `wk-ios`, `qualiow-doctor-mobile` and `qualiow-setup-mobile` on your `PATH`.
|
|
12
|
+
|
|
9
13
|
## TL;DR
|
|
10
14
|
|
|
11
15
|
```bash
|
|
12
16
|
# 1. Install everything scriptable + create standard devices (idempotent, re-runnable)
|
|
13
|
-
|
|
17
|
+
bin/setup-mobile.sh # or --android / --ios; --yes for non-interactive
|
|
14
18
|
|
|
15
19
|
# 2. Confirm the machine is ready (read-only; installs nothing)
|
|
16
|
-
|
|
20
|
+
bin/doctor-mobile.sh # add --quiet to print only failures + the verdict
|
|
17
21
|
|
|
18
22
|
# 3. Run a mobile session (once doctor says READY)
|
|
19
|
-
|
|
20
|
-
# drive via bin/mcli — see the target config notes in data/targets/_example-sim-*.yml
|
|
23
|
+
/qa-explore-mobile --target my-mobile-target
|
|
21
24
|
```
|
|
22
25
|
|
|
23
|
-
`
|
|
24
|
-
|
|
26
|
+
`bin/setup-mobile.sh` automates everything it can and **prints clear MANUAL steps** for the
|
|
27
|
+
few things it can't (Xcode + the iOS runtime). It finishes by running the doctor.
|
|
25
28
|
|
|
26
29
|
## What gets installed / created
|
|
27
30
|
|
|
28
31
|
| Component | Needed for | Auto by `setup-mobile.sh`? |
|
|
29
32
|
|-----------|-----------|----------------------------|
|
|
30
|
-
| Homebrew | base | ✅ (offers to install) |
|
|
31
|
-
| Node ≥
|
|
32
|
-
| **Maestro** | the driver (hierarchy/tap/input) | ✅ (`get.maestro.mobile.dev`) |
|
|
33
|
-
| **JDK 17** | Android SDK tooling | ✅ `brew install openjdk@17` |
|
|
34
|
-
| Android cmdline-tools | `sdkmanager`/`avdmanager` | ✅ `brew install --cask android-
|
|
33
|
+
| Homebrew | base | ✅ (offers to install; `curl \| bash`) |
|
|
34
|
+
| **Node ≥ 22.4** | runs `mobile-cli.mjs`, `wkeval.mjs` | ✅ |
|
|
35
|
+
| **Maestro** (pinned, default 2.10.0) | the driver (hierarchy/tap/input) | ✅ (`get.maestro.mobile.dev`, `curl \| bash`) |
|
|
36
|
+
| **JDK 17** | Maestro and the Android SDK tooling | ✅ `brew install openjdk@17` |
|
|
37
|
+
| Android cmdline-tools | `sdkmanager`/`avdmanager` | ✅ `brew install --cask android-commandlinetools` |
|
|
35
38
|
| platform-tools (`adb`), `emulator` | drive Android | ✅ `sdkmanager` |
|
|
36
39
|
| Google **Play** system image (API 35) | **Chrome must be present** | ✅ `sdkmanager` |
|
|
37
40
|
| SDK licenses | unblock installs | ✅ `sdkmanager --licenses` |
|
|
@@ -39,10 +42,31 @@ for the few things it can't (Xcode + the iOS runtime). It finishes by running th
|
|
|
39
42
|
| **Xcode** + CLI tools | iOS simulators at all | ⚠️ **manual** (App Store, multi-GB) |
|
|
40
43
|
| iOS runtime | a simulator OS to run | ⚠️ **manual** if none installed (Xcode ▸ Settings ▸ Components) |
|
|
41
44
|
| iOS simulator device `qa-iphone` | a device to boot | ✅ `simctl create` (once Xcode + a runtime exist) |
|
|
45
|
+
| `ios-webkit-debug-proxy` + `python3` | the `bin/wk-ios` DOM bridge (iOS Safari) | ✅ `brew install ios-webkit-debug-proxy` (python3 warned if absent) |
|
|
42
46
|
| Safari / Chrome (the browsers) | web-mode SUT | ✅ preinstalled (Safari always; Chrome via the Play image) |
|
|
43
47
|
|
|
48
|
+
### Flags
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
bin/setup-mobile.sh # both platforms, interactive
|
|
52
|
+
bin/setup-mobile.sh --android # scope to one platform
|
|
53
|
+
bin/setup-mobile.sh --ios
|
|
54
|
+
bin/setup-mobile.sh --yes # non-interactive: assume yes to prompts
|
|
55
|
+
bin/setup-mobile.sh --yes --allow-curl-bash # ...and let the curl|bash installers run too
|
|
56
|
+
MAESTRO_VERSION=2.10.0 bin/setup-mobile.sh # pin a specific Maestro release
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The two `curl | bash` installers (Homebrew and Maestro) **always ask first**. Under `--yes`
|
|
60
|
+
they are *skipped* with the exact command printed, unless `--allow-curl-bash` is also given —
|
|
61
|
+
so an unattended run never pipes a remote script into a shell behind your back.
|
|
62
|
+
|
|
63
|
+
Other environment overrides: `ANDROID_API` (default 35), `ANDROID_AVD_NAME`, `IOS_DEVICE_NAME`,
|
|
64
|
+
`ANDROID_HOME`.
|
|
65
|
+
|
|
44
66
|
### The two genuinely manual iOS steps
|
|
67
|
+
|
|
45
68
|
Xcode can't be reliably scripted:
|
|
69
|
+
|
|
46
70
|
```bash
|
|
47
71
|
# After installing Xcode from the App Store:
|
|
48
72
|
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
|
|
@@ -50,9 +74,32 @@ sudo xcodebuild -license accept
|
|
|
50
74
|
# If no iOS runtime is installed:
|
|
51
75
|
xcodebuild -downloadPlatform iOS # or Xcode ▸ Settings ▸ Components
|
|
52
76
|
```
|
|
53
|
-
|
|
77
|
+
|
|
78
|
+
Then re-run `bin/setup-mobile.sh` — it will create the `qa-iphone` device.
|
|
79
|
+
|
|
80
|
+
## The preflight
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
bin/doctor-mobile.sh # check both platforms
|
|
84
|
+
bin/doctor-mobile.sh --ios # iOS only
|
|
85
|
+
bin/doctor-mobile.sh --android # Android only
|
|
86
|
+
bin/doctor-mobile.sh --quiet # only failures + the final verdict
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Read-only: it installs nothing and prints `✓`/`✗` with the exact fix command for each gap.
|
|
90
|
+
The mobile session's `00-setup` phase runs it first, so a fresh machine fails fast with
|
|
91
|
+
actionable guidance instead of a cryptic error deep in a session. Exit code 0 means every
|
|
92
|
+
REQUIRED check for the requested platform passed.
|
|
93
|
+
|
|
94
|
+
It checks macOS, Homebrew, **Node ≥ 22.4**, **Maestro ≥ 2.6.0** (`MAESTRO_MIN_VERSION` —
|
|
95
|
+
`mobile-cli` parses the JSON-only `maestro hierarchy` introduced in 2.6), then per platform:
|
|
96
|
+
JDK 17, the SDK root, `adb`, `emulator`, `sdkmanager`/`avdmanager` and an AVD for Android;
|
|
97
|
+
Xcode CLT + full Xcode, an iOS runtime and an iPhone simulator for iOS. `ios-webkit-debug-proxy`
|
|
98
|
+
and `python3` are reported as **warnings** — without them the `bin/wk-ios` DOM bridge is
|
|
99
|
+
unavailable, but a session can still run.
|
|
54
100
|
|
|
55
101
|
## Standard device names (referenced by the target configs)
|
|
102
|
+
|
|
56
103
|
- Android AVD: **`qa_pixel_api35`** (Google-Play API 35) — set `ANDROID_AVD_NAME` to override.
|
|
57
104
|
- iOS sim: **`qa-iphone`** — set `IOS_DEVICE_NAME` to override.
|
|
58
105
|
|
|
@@ -60,15 +107,92 @@ The `data/targets/_example-sim-*.yml` configs resolve the device **by name** (no
|
|
|
60
107
|
hardcoded UDID) so they're portable across machines. `00-setup` boots the named device,
|
|
61
108
|
creating it if missing.
|
|
62
109
|
|
|
110
|
+
## Driving the device
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
bin/mcli boot qa_pixel_api35 # boots an AVD or an iOS sim by name, waits, set-devices it
|
|
114
|
+
bin/mcli set-device emulator-5554 --platform android # adb serial, iOS UDID, or iOS sim NAME
|
|
115
|
+
bin/mcli set-app com.android.chrome # or com.apple.mobilesafari, or your app id
|
|
116
|
+
bin/mcli launch
|
|
117
|
+
bin/mcli open-url https://staging.m.example.com # web mode (alias of deep-link)
|
|
118
|
+
bin/mcli snapshot # a11y tree as refs e1, e2, … — refs expire after 60 s
|
|
119
|
+
bin/mcli fill e4 my-username
|
|
120
|
+
bin/mcli snapshot # ALWAYS re-snapshot after the keyboard appears
|
|
121
|
+
bin/mcli click e7
|
|
122
|
+
bin/mcli info # resolved device/app + which state file is in use
|
|
123
|
+
bin/mcli --help # the full command surface
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
`set-device` resolves the platform by matching the id against `xcrun simctl list devices` and
|
|
127
|
+
`adb devices`; pass `--platform ios|android` when it cannot tell. `boot` handles the
|
|
128
|
+
`emulator -avd …` / `simctl boot && open -a Simulator` dance for you.
|
|
129
|
+
|
|
130
|
+
**State.** Device and app selection is stored in a small JSON file. `--state <file>` (or
|
|
131
|
+
`$MOBILE_CLI_STATE`) picks it explicitly; otherwise `set-device` chooses
|
|
132
|
+
`~/.mobile-cli/<device>.json`. Give each concurrent session its own state file:
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
bin/mcli --state /tmp/run-a.json set-device qa-iphone
|
|
136
|
+
bin/mcli --state /tmp/run-a.json set-app com.apple.mobilesafari
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
**Exit codes.** `0` ok · `1` command failed · `2` usage · `3` stale or unknown ref (re-run
|
|
140
|
+
`snapshot`) · `4` no device or app set. A stale ref is not a flake: the tree is renumbered on
|
|
141
|
+
every snapshot, so a ref older than `MCLI_REF_TTL_MS` (default 60 000) is rejected rather than
|
|
142
|
+
tapped at a coordinate that has since moved.
|
|
143
|
+
|
|
144
|
+
**Evidence.** `screenshot <path>` needs no app. `logs [--since <sec>] [--errors]` reads logcat
|
|
145
|
+
or `simctl log`; `logs-clear` empties the Android buffer. `record-start <path>.mp4` /
|
|
146
|
+
`record-stop` produce mp4 on both platforms (Android caps a single recording at 180 s).
|
|
147
|
+
|
|
148
|
+
## The helper wrappers
|
|
149
|
+
|
|
150
|
+
`bin/mcli`, `bin/wadb` and `bin/wk-ios` exist because Claude Code permission rules match on
|
|
151
|
+
the **first token** of a Bash command — a compound `export JAVA_HOME=…; node …` line can never
|
|
152
|
+
be pre-approved. Each wrapper resolves the environment internally and then execs the real tool.
|
|
153
|
+
|
|
154
|
+
### `bin/wadb` — raw adb, environment resolved
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
bin/wadb devices
|
|
158
|
+
bin/wadb logcat -d | tail -50
|
|
159
|
+
bin/wadb shell pm list packages | grep chrome
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
It uses the `adb` already on `PATH` when there is one, else the SDK copy under `ANDROID_HOME`
|
|
163
|
+
(default `~/Library/Android/sdk`).
|
|
164
|
+
|
|
165
|
+
### `bin/wk-ios` — real JS/DOM in iOS Simulator Safari
|
|
166
|
+
|
|
167
|
+
Maestro reads the native XCUITest accessibility tree, which collapses Safari's web content
|
|
168
|
+
into unlabeled nodes. The DOM itself is fully addressable through WebKit's Remote Inspector,
|
|
169
|
+
and `wk-ios` wires `ios-webkit-debug-proxy` up to the running Simulator so you can evaluate JS
|
|
170
|
+
in the frontmost page — the same power `playwright-cli eval` gives you on desktop, but on the
|
|
171
|
+
real iOS engine.
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
bin/wk-ios 'document.title'
|
|
175
|
+
bin/wk-ios 'document.querySelectorAll(".cart_item").length'
|
|
176
|
+
bin/wk-ios --url # print the page WebSocket URL
|
|
177
|
+
bin/wk-ios --stop # stop the proxy this script started (never anyone else's)
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Requires `ios-webkit-debug-proxy`, `python3`, Node ≥ 22.4 (`wkeval.mjs` uses the global
|
|
181
|
+
`WebSocket`), and a booted Simulator with Safari on a page. `WK_IOS_PORT` (default 9221) sets
|
|
182
|
+
the iwdp device-list port; pages are served on `WK_IOS_PORT + 1`.
|
|
183
|
+
|
|
63
184
|
## Why a Google **Play** (or `google_apis`) Android image
|
|
185
|
+
|
|
64
186
|
A stock **AOSP** system image ships **no `com.android.chrome`** — only a basic AOSP
|
|
65
187
|
browser or none. Web-mode Android targets need Chrome, so `setup-mobile.sh` installs a
|
|
66
188
|
`google_apis_playstore` image. Verify on any AVD with:
|
|
189
|
+
|
|
67
190
|
```bash
|
|
68
|
-
|
|
191
|
+
bin/wadb shell pm list packages | grep chrome
|
|
69
192
|
```
|
|
70
193
|
|
|
71
194
|
## Auth reality (web targets)
|
|
195
|
+
|
|
72
196
|
There is **no transferable `storage_state`** for a real device browser (unlike Playwright).
|
|
73
197
|
SSO + MFA logins are done **once, interactively, by a human** in the sim/emulator browser;
|
|
74
198
|
the device persists the browser profile across launches. Never script MFA. Never
|
|
@@ -76,17 +200,24 @@ the device persists the browser profile across launches. Never script MFA. Never
|
|
|
76
200
|
`notes:` blocks in `data/targets/_example-sim-*.yml`.
|
|
77
201
|
|
|
78
202
|
## Troubleshooting
|
|
203
|
+
|
|
79
204
|
- **`doctor-mobile.sh` says NOT READY** → fix each `✗` with the printed command, or run
|
|
80
205
|
`setup-mobile.sh`, then re-run the doctor.
|
|
81
206
|
- **`JAVA_HOME` / `adb` not found mid-session** → `bin/mcli` resolves JDK 17 via
|
|
82
|
-
`/usr/libexec/java_home -v 17`
|
|
83
|
-
Export them explicitly if your SDK lives
|
|
207
|
+
`/usr/libexec/java_home -v 17` (falling back to the Homebrew `openjdk@17` prefixes) and
|
|
208
|
+
defaults `ANDROID_HOME` to `~/Library/Android/sdk`. Export them explicitly if your SDK lives
|
|
209
|
+
elsewhere.
|
|
210
|
+
- **Maestro too old** → `mobile-cli` needs the JSON-only `maestro hierarchy` of 2.6.0+.
|
|
211
|
+
Reinstall pinned: `curl -Ls https://get.maestro.mobile.dev | MAESTRO_VERSION=2.10.0 bash`.
|
|
212
|
+
- **Every tap hits the wrong element** → your refs expired. Re-run `snapshot`; exit code 3
|
|
213
|
+
means exactly this.
|
|
84
214
|
- **Intel Mac** → `setup-mobile.sh` selects `x86_64` system images automatically (`arm64`
|
|
85
215
|
Macs get `arm64-v8a`).
|
|
86
216
|
- **Emulator slow/flaky to boot or Chrome absent** → fall back to the iOS Safari target;
|
|
87
217
|
both engines are independent.
|
|
88
218
|
|
|
89
219
|
## Scope
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
220
|
+
|
|
221
|
+
This covers iOS Simulator + Android Emulator (browser **and** native modes) on macOS. It does
|
|
222
|
+
**not** provision physical devices, CI runners, Windows/Linux hosts, or the desktop web path
|
|
223
|
+
(`/qa-explore`, which only needs `playwright-cli`).
|