qualiow-exploratory-testing 1.2.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (111) hide show
  1. package/.claude-plugin/plugin.json +22 -0
  2. package/.env.example +27 -0
  3. package/CHANGELOG.md +194 -0
  4. package/README.md +428 -179
  5. package/agents/qa-gather-agent.md +24 -0
  6. package/{scripts → bin}/doctor-mobile.sh +57 -18
  7. package/bin/mcli +15 -9
  8. package/bin/mobile-cli.mjs +574 -286
  9. package/{scripts → bin}/setup-mobile.sh +56 -19
  10. package/bin/wadb +10 -4
  11. package/bin/wk-ios +29 -17
  12. package/bin/wkeval.mjs +7 -1
  13. package/data/domains/_default.yml +1 -1
  14. package/data/domains/identity.yml +138 -0
  15. package/data/knowledge/changelog.yml +104 -0
  16. package/data/knowledge/learned-patterns.md +172 -86
  17. package/data/knowledge/manifest.yml +481 -70
  18. package/data/knowledge/releases/v0.1.0/entries/technique-business-logic-race-conditions.yml +1 -1
  19. package/data/knowledge/releases/v0.1.0/release.yml +8 -1
  20. package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
  21. package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
  22. package/data/knowledge/releases/v0.2.0/release.yml +20 -0
  23. package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
  24. package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
  25. package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
  26. package/data/knowledge/releases/v0.3.0/release.yml +25 -0
  27. package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
  28. package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
  29. package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
  30. package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
  31. package/data/knowledge/releases/v0.4.0/release.yml +28 -0
  32. package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
  33. package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
  34. package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
  35. package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
  36. package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
  37. package/data/knowledge/releases/v0.5.0/release.yml +33 -0
  38. package/data/knowledge/releases/v0.6.0/entries/technique-async-callback-contracts.yml +98 -0
  39. package/data/knowledge/releases/v0.6.0/entries/technique-exactly-once-verification.yml +116 -0
  40. package/data/knowledge/releases/v0.6.0/release.yml +30 -0
  41. package/data/security/SECURITY-POLICY.md +273 -0
  42. package/data/targets/_default.yml +1 -1
  43. package/data/targets/_example-api-only.yml +80 -0
  44. package/data/targets/_example-backend.yml +123 -0
  45. package/data/targets/_example-mobile-emulation.yml +2 -2
  46. package/data/targets/_example-native-mobile.yml +2 -2
  47. package/data/targets/_example-sim-android-chrome.yml +1 -1
  48. package/data/targets/_example-sim-ios-safari.yml +1 -1
  49. package/data/templates/api-probe-matrix.md +99 -0
  50. package/data/templates/bug-report.md +29 -151
  51. package/data/templates/coverage-map.md +23 -6
  52. package/data/templates/expected-behaviour.md +94 -0
  53. package/data/templates/session-charter.md +3 -0
  54. package/data/templates/session-report.md +57 -80
  55. package/dist/cli/index.js +1500 -739
  56. package/dist/index.d.ts +673 -75
  57. package/dist/index.js +1664 -270
  58. package/docs/BACKEND-VERIFICATION.md +307 -0
  59. package/docs/GETTING-STARTED.md +188 -128
  60. package/docs/MOBILE-SETUP.md +149 -18
  61. package/package.json +32 -16
  62. package/skills/qa-explore/SKILL.md +31 -25
  63. package/skills/qa-explore/phases/00-setup.md +67 -41
  64. package/skills/qa-explore/phases/01-auth.md +32 -15
  65. package/skills/qa-explore/phases/02-charter.md +16 -5
  66. package/skills/qa-explore/phases/03-discovery.md +50 -26
  67. package/skills/qa-explore/phases/04-journeys.md +2 -2
  68. package/skills/qa-explore/phases/05-features.md +39 -24
  69. package/skills/qa-explore/phases/06-edge-cases.md +24 -13
  70. package/skills/qa-explore/phases/07-reporting.md +79 -140
  71. package/skills/qa-explore/references/context-integration.md +1 -1
  72. package/skills/qa-explore/references/output-contract.md +187 -0
  73. package/skills/qa-explore/references/paths.md +84 -0
  74. package/skills/qa-explore/references/playwright-agents-integration.md +61 -84
  75. package/skills/qa-explore/references/security-rules.md +66 -25
  76. package/skills/qa-explore/references/severity-guide.md +31 -4
  77. package/skills/qa-explore-cleanup/SKILL.md +17 -6
  78. package/skills/qa-explore-feedback/SKILL.md +18 -5
  79. package/skills/qa-explore-mobile/SKILL.md +86 -64
  80. package/skills/qa-explore-mobile/phases/00-setup.md +71 -62
  81. package/skills/qa-explore-mobile/phases/01-auth.md +28 -31
  82. package/skills/qa-explore-mobile/phases/02-charter.md +8 -4
  83. package/skills/qa-explore-mobile/phases/03-discovery.md +17 -11
  84. package/skills/qa-explore-mobile/phases/04-journeys.md +6 -6
  85. package/skills/qa-explore-mobile/phases/05-features.md +9 -9
  86. package/skills/qa-explore-mobile/phases/06-edge-cases.md +35 -40
  87. package/skills/qa-explore-mobile/phases/07-reporting.md +82 -56
  88. package/skills/qa-explore-mobile/references/mobile-edges.md +26 -26
  89. package/skills/qa-explore-quick/SKILL.md +60 -25
  90. package/skills/qa-explore-report/SKILL.md +40 -34
  91. package/skills/qa-gather/SKILL.md +17 -2
  92. package/skills/qa-knowledge-add/SKILL.md +20 -9
  93. package/skills/qa-knowledge-list/SKILL.md +12 -3
  94. package/skills/qa-target-setup/SKILL.md +63 -41
  95. package/skills/qa-verify-backend/SKILL.md +172 -0
  96. package/skills/qa-verify-backend/phases/00-setup.md +142 -0
  97. package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
  98. package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
  99. package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
  100. package/skills/qa-verify-backend/phases/03b-api-verification.md +170 -0
  101. package/skills/qa-verify-backend/phases/04-e2e-trigger.md +82 -0
  102. package/skills/qa-verify-backend/phases/05-reporting.md +163 -0
  103. package/skills/qa-verify-backend/references/api-probes.md +168 -0
  104. package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
  105. package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
  106. package/skills/qa-verify-backend/references/payload-verification.md +112 -0
  107. package/skills/qa-verify-backend/references/release-readiness.md +97 -0
  108. package/skills/qa-verify-backend/references/safety-rules.md +95 -0
  109. package/dist/cli/index.d.ts +0 -1
  110. package/dist/cli/index.js.map +0 -1
  111. package/dist/index.js.map +0 -1
package/README.md CHANGED
@@ -1,299 +1,548 @@
1
- # @qualiow/exploratory-testing
1
+ # qualiow-exploratory-testing
2
2
 
3
- qualiow -- AI-powered exploratory testing skills using Claude Code + Playwright CLI.
3
+ AI-powered exploratory testing for **web**, **mobile** and **backend/API** — Claude Code skills, a QA knowledge base, and the CLI that wires them into a project.
4
4
 
5
- Claude acts as a Principal QA Engineer: it explores web applications systematically, finds bugs, and generates structured reports. No test scripts, no framework -- just AI reasoning, browser commands, and QA heuristics.
5
+ Claude acts as a Principal QA Engineer: it understands the business first, risk-ranks what
6
+ matters, explores systematically, and writes bug reports an executive can act on. No test
7
+ scripts and no framework — AI reasoning, browser/device commands, and QA heuristics.
6
8
 
7
- ## How to Install This Project
9
+ | Surface | Driver | Skill |
10
+ |---------|--------|-------|
11
+ | Web app in a desktop browser | `playwright-cli` | `/qa-explore`, `/qa-explore-quick` |
12
+ | Native iOS/Android app, or a web app in **real** Simulator Safari / Emulator Chrome | `bin/mobile-cli.mjs` (Maestro shim) | `/qa-explore-mobile` |
13
+ | Acceptance criteria with no UI at all — tables, streams, queues, IAM, HTTP endpoints | `git show`, read-only `aws-cli`, authenticated `fetch` | `/qa-verify-backend` |
8
14
 
9
- ### Prerequisites
15
+ ## Requirements
10
16
 
11
- - Node.js >= 18
12
- - Playwright CLI (`npm install @playwright/cli`)
13
- - Claude Code (for running exploratory sessions)
17
+ - **Node.js >= 22.4** (`node --version`)
18
+ - **Claude Code** — the skills run as slash commands inside it
19
+ - `playwright-cli` ships as a dependency of this package; nothing extra to install for the
20
+ web path. Mobile needs the iOS/Android toolchains — see [docs/MOBILE-SETUP.md](docs/MOBILE-SETUP.md).
14
21
 
15
- ### Quick Start
22
+ ## Install
16
23
 
17
- ```bash
18
- # install globally
19
- npm install -g @qualiow/exploratory-testing
24
+ Three supported paths. Pick one.
25
+
26
+ ### 1. npm (recommended for a project)
20
27
 
21
- # initialize in your project
28
+ ```bash
29
+ npm install -g qualiow-exploratory-testing
30
+ cd my-project
22
31
  qualiow init
32
+ ```
33
+
34
+ `qualiow init` copies into the current project:
23
35
 
24
- # run your first session
25
- /qa-explore https://your-app.com
36
+ | Copied | To |
37
+ |--------|----|
38
+ | the 11 skills | `.claude/skills/qa-*/` |
39
+ | the `qa-gather-agent` sub-agent | `.claude/agents/` |
40
+ | knowledge base, domain profiles (`*.yml`), templates, security policy | `data/` |
41
+ | `_default.yml` target (with `--include-examples`: the `_example-*.yml` templates and `testers-ai.yml`) | `data/targets/` |
42
+ | the mobile driver, `setup-mobile.sh`, `doctor-mobile.sh` (executable) | `qa/bin/` |
43
+ | the credential template | `qa/.env.example` |
44
+
45
+ It also creates `output/{sessions,bugs,context}`, `.auth/` and `qa/`, and appends a
46
+ `# Qualiow` block to `.gitignore` (`.auth/`, `.env`, `qa/.env`, `output/sessions/*/`,
47
+ `!output/sessions/INDEX.md`, `output/context/*.md`, `data/targets/local-*.yml`,
48
+ `.playwright-cli/`, `*.trace.zip`, `*.webm`).
49
+
50
+ Re-running `init` is a no-op — existing files are kept unless you pass `--force`, and
51
+ `--dry-run` prints the plan without writing anything.
52
+
53
+ ```bash
54
+ cp qa/.env.example qa/.env # credentials live here, never in YAML
55
+ npx playwright-cli install --skills # optional: the official Playwright skill alongside
56
+ /qa-explore https://your-app.com # inside Claude Code
26
57
  ```
27
58
 
28
- ### Local Development
59
+ ### 2. Claude Code plugin
60
+
61
+ ```bash
62
+ claude --plugin-dir /path/to/qualiow-exploratory-testing-skills
63
+ ```
64
+
65
+ Skills appear namespaced: `/qualiow:qa-explore`, `/qualiow:qa-verify-backend`, and so on.
66
+ Data (knowledge, domains, templates, targets) resolves from `${CLAUDE_PLUGIN_ROOT}/data/`
67
+ unless the project you are in has its own `data/` or `qa/` directory, which wins.
68
+
69
+ `bin/` lands on `PATH` only for **marketplace** installs — with `--plugin-dir`, call the
70
+ driver by its full path, `${CLAUDE_PLUGIN_ROOT}/bin/mcli`.
71
+
72
+ ### 3. Git checkout (contributors, dogfooding)
29
73
 
30
74
  ```bash
31
- # clone the repository
32
75
  git clone https://github.com/willcoliveira/qualiow-exploratory-testing-skills.git
33
76
  cd qualiow-exploratory-testing-skills
34
-
35
- # install dependencies
36
77
  npm install
37
-
38
- # build
39
78
  npm run build
40
-
41
- # run tests
42
79
  npm test
43
80
  ```
44
81
 
45
- ## How to Run the CLI
82
+ `.claude/skills/` and `.claude/agents/` are canonical. `skills/` and `agents/` are generated
83
+ mirrors (shipped to npm and used by the plugin) kept in sync by `npm run sync:plugin`; CI
84
+ fails on drift, so never hand-edit the mirrors.
46
85
 
47
- | Command | Description |
48
- |---------|-------------|
49
- | `qualiow init` | Initialize qualiow in the current project |
50
- | `qualiow explore [url]` | Run an exploratory testing session |
51
- | `qualiow validate` | Validate all configuration files (targets, knowledge, domains) |
52
- | `qualiow report` | Generate a report from an existing session |
53
- | `qualiow list <type>` | List sessions, knowledge, targets, or domains |
54
- | `qualiow gather` | Gather and analyze requirements for testing context |
86
+ ## CLI
55
87
 
56
- ## How to Run Exploratory Sessions
88
+ The CLI is the plumbing around the skills — project setup, config validation, session
89
+ listing, report export. **The exploratory sessions themselves run inside Claude Code**, not
90
+ from the shell.
57
91
 
58
- ### Full session (45 minutes)
92
+ | Command | Flags | What it does |
93
+ |---------|-------|--------------|
94
+ | `qualiow init` | `--include-examples` `--force` `--dry-run` | Install skills, agent, data, `qa/bin/` and the output tree into the current project. Idempotent |
95
+ | `qualiow explore [url]` | `-t <target>` `-c <context>` `--time-box 45m` `--dry-run` | **Pre-flight only.** Validates the inputs, creates the session directory skeleton, and prints the `/qa-explore … --session <dir>` command to paste into Claude Code. It does not drive a browser |
96
+ | `qualiow validate` | `--targets` `--domains` `--knowledge` `--kb` `--all` | Validate configs against their schemas; exits 1 on any failure. `--all` also checks a project-local `qa/target.yml` and cross-checks the knowledge base against its manifest |
97
+ | `qualiow list <type>` | `sessions` \| `knowledge` \| `targets` \| `domains` | List what is installed or recorded |
98
+ | `qualiow report` | `-s <id>\|latest` `-f md\|html\|json\|jira` `-o <file>` `--stdout` | Export an existing session. `md` writes `session-summary.md`; html/json/jira carry the confidentiality header and pass through redaction |
99
+ | `qualiow kb <sync\|check>` | — | Regenerate or validate `data/knowledge/manifest.yml` from the release files |
100
+ | `qualiow --version` | — | Print the package version |
59
101
 
60
- ```bash
61
- # explore a public site by URL
62
- /qa-explore https://testers.ai/testing/
63
- ```
102
+ `qualiow gather` was **removed in 2.0.0** — it was a stub that wrote nothing. Use the
103
+ `/qa-gather` skill, which does the real work.
104
+
105
+ A global install also puts the mobile helpers on `PATH`: `mcli`, `wadb`, `wk-ios`,
106
+ `qualiow-doctor-mobile`, `qualiow-setup-mobile`.
107
+
108
+ ## Running sessions
109
+
110
+ ### Full session (45 minutes, 8 phases)
64
111
 
65
112
  ```bash
66
- # explore using a saved target configuration
67
- /qa-explore --target company-staging
113
+ /qa-explore https://testers.ai/testing/ # any public site
114
+ /qa-explore --target company-staging # a saved target config
115
+ /qa-explore --target company-staging --context output/context/TICKET-123-context.md
68
116
  ```
69
117
 
70
- ### Quick focused session (15 minutes)
118
+ Setup 2 min · Auth 2 · Charter 5 · Discovery 6 · Journeys 10 · Features 10 · Edge cases 6 ·
119
+ Reporting 4. Findings are written to disk between phases so context never overflows.
120
+
121
+ ### Quick session (15 minutes, one page or feature)
71
122
 
72
123
  ```bash
73
- # check a single page or feature
74
124
  /qa-explore-quick https://app.example.com/checkout
75
125
  ```
76
126
 
77
- ### Mobile session (simulator/emulator)
127
+ Quick sessions produce the same artefacts as a full one — `session-report.md`,
128
+ `bugs/BUG-NNN.md`, `stats.json`, an `INDEX.md` row — so every downstream skill can read them.
129
+
130
+ ### Gather requirements first
131
+
132
+ ```bash
133
+ /qa-gather # files, URLs, tickets, pasted text
134
+ ```
78
135
 
79
- Runs a full exploratory session on an iOS Simulator or Android Emulator — either a
80
- **native app** (the installed app is the system under test) or a **mobile web app in the
81
- real device browser** (iOS Simulator Safari / Android Emulator Chrome — the genuine
82
- engines, which Playwright cannot drive). The mode comes from the target config.
136
+ Writes `output/context/<TICKET>-context.md`, which `/qa-explore` and `/qa-verify-backend`
137
+ consume via `--context`.
83
138
 
84
- #### 1. One-time machine setup
139
+ ### After a session
85
140
 
86
141
  ```bash
87
- # installs everything scriptable: Maestro, JDK 17, Android SDK + Google-Play AVD
88
- # (qa_pixel_api35), ios-webkit-debug-proxy, and the qa-iphone simulator device
89
- scripts/setup-mobile.sh # add --android or --ios to scope; --yes for non-interactive
142
+ /qa-explore-report output/sessions/2026-09-08-1813-explore-parabank # regenerate/reformat
143
+ /qa-explore-feedback # false positives, missed bugs
144
+ /qa-explore-cleanup # archive or delete old sessions
145
+ /qa-target-setup # configure a new target
146
+ /qa-knowledge-add # teach it a new heuristic
147
+ /qa-knowledge-list # browse the knowledge base
148
+ ```
149
+
150
+ ### What a session leaves behind
151
+
152
+ Every session — web, quick, mobile or backend — lands in
153
+ `output/sessions/<YYYY-MM-DD-HHmm>-<kind>-<slug>/`, where `kind` is `explore`, `quick`,
154
+ `mobile` or `backend`:
155
+
156
+ ```
157
+ output/sessions/2026-09-08-1813-explore-parabank/
158
+ charter.md # what is being tested and why
159
+ session-log.md # real-time findings with WHY reasoning
160
+ phase-3-discovery.md # phase artefacts (file number == phase number)
161
+ phase-4-journeys.md
162
+ phase-5-features.md
163
+ phase-6-edge-cases.md
164
+ bugs/BUG-001.md # one bug = one report
165
+ screenshots/BUG-001.png
166
+ videos/
167
+ session-report.md # executive summary, coverage map, recommendations
168
+ stats.json
169
+ ```
170
+
171
+ Plus a row in `output/sessions/INDEX.md`
172
+ (`| Date | Kind | Target | Bugs | Duration | Status | Report |`) and an entry in
173
+ `output/bugs/all-bugs.md`. Bug reports open with the two-line confidentiality blockquote and
174
+ the headline `# BUG-NNN: [Component] fails [Condition] causing [Impact]`.
175
+
176
+ ### Project-local configuration
177
+
178
+ `/qa-target-setup` writes `qa/target.yml` (project-local, one target per repo) or, with
179
+ `--shared`, `data/targets/<id>.yml`. Resolution order is:
180
+
181
+ ```
182
+ --target <name> → qa/target.yml → data/targets/_default.yml
90
183
  ```
91
184
 
92
- Two iOS steps are genuinely manual (the script prints them if needed):
185
+ Credentials come from `qa/.env`, falling back to `.env`. The env var names the shipped
186
+ configs reference: `QA_USER`, `QA_PASS`, `QA_TOKEN`, `QA_AWS_PROFILE`, `QA_AWS_REGION`,
187
+ `QA_API_TOKEN`, `EXAMPLE_API_KEY`. Never put a secret in a YAML file — reference it by
188
+ variable name.
189
+
190
+ ## Mobile sessions
191
+
192
+ `/qa-explore-mobile` runs a full exploratory session on an iOS Simulator or Android
193
+ Emulator — either a **native app** (the installed app is the system under test) or a **web
194
+ app in the real device browser** (Simulator Safari / Emulator Chrome — the genuine engines,
195
+ which Playwright cannot drive). The mode comes from the target config.
196
+
197
+ ### 1. One-time machine setup
93
198
 
94
199
  ```bash
95
- # a) install Xcode from the App Store, then:
96
- sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
97
- sudo xcodebuild -license accept
98
- # b) if no iOS runtime is installed:
99
- xcodebuild -downloadPlatform iOS
100
- # then re-run scripts/setup-mobile.sh --ios (creates the qa-iphone simulator)
200
+ bin/setup-mobile.sh # both platforms
201
+ bin/setup-mobile.sh --android # or --ios to scope
202
+ bin/setup-mobile.sh --yes --allow-curl-bash # non-interactive, incl. the curl|bash installers
101
203
  ```
102
204
 
103
- #### 2. Verify readiness (read-only, run before every first session on a machine)
205
+ It installs everything scriptable — Maestro (pinned via `MAESTRO_VERSION`, default 2.10.0),
206
+ JDK 17, Android cmdline-tools + platform-tools + a Google-Play system image, the standard AVD
207
+ `qa_pixel_api35`, the `qa-iphone` simulator device, and `ios-webkit-debug-proxy` — and prints
208
+ the two genuinely manual iOS steps (installing Xcode, downloading an iOS runtime).
209
+
210
+ ### 2. Preflight
104
211
 
105
212
  ```bash
106
- scripts/doctor-mobile.sh # or --ios / --android; fix each ✗ until READY
213
+ bin/doctor-mobile.sh # or --ios / --android / --quiet
107
214
  ```
108
215
 
109
- #### 3. Create a target and run
216
+ Read-only. Installs nothing; prints `✓`/`✗` with the exact fix command for each gap, and
217
+ checks Node >= 22.4, python3, Maestro >= 2.6.0 and `ios-webkit-debug-proxy`.
110
218
 
111
- Copy a template from `data/targets/`, rename it, and point it at your app:
219
+ ### 3. Pick a target template
220
+
221
+ ```bash
222
+ cp data/targets/_example-sim-ios-safari.yml data/targets/local-my-app.yml
223
+ ```
112
224
 
113
225
  | Template | What it tests | Driver |
114
226
  |----------|---------------|--------|
115
227
  | `_example-native-mobile.yml` | An installed iOS/Android app (native mode) | mobile-cli (Maestro) |
116
- | `_example-sim-ios-safari.yml` | A mobile web app in REAL iOS Simulator Safari | mobile-cli (Maestro) |
117
- | `_example-sim-android-chrome.yml` | A mobile web app in REAL Android Emulator Chrome | mobile-cli (Maestro) |
228
+ | `_example-sim-ios-safari.yml` | A mobile web app in **real** iOS Simulator Safari | mobile-cli (Maestro) |
229
+ | `_example-sim-android-chrome.yml` | A mobile web app in **real** Android Emulator Chrome | mobile-cli (Maestro) |
118
230
  | `_example-mobile-emulation.yml` | A mobile web app via Playwright device emulation (no simulator) | `/qa-explore` (playwright-cli) |
119
231
 
120
232
  ```bash
121
- /qa-explore-mobile --target my-mobile-target # real sim/emulator (native or web)
122
- /qa-explore --target my-emulation-target # Playwright mobile emulation
233
+ /qa-explore-mobile --target local-my-app # real sim/emulator, native or web
234
+ /qa-explore --target local-my-emulation # Playwright mobile emulation
123
235
  ```
124
236
 
125
- #### 4. Driving the device manually (what the skill does under the hood)
237
+ ### 4. Driving the device by hand
126
238
 
127
- ```bash
128
- # Android: boot the standard AVD # iOS: boot the standard simulator
129
- ~/Library/Android/sdk/emulator/emulator \
130
- -avd qa_pixel_api35 -no-snapshot & xcrun simctl boot qa-iphone && open -a Simulator
239
+ The skill runs `bin/mcli` (or `qa/bin/mcli` in an init'ed project, or `mcli` on `PATH`).
240
+ Everything it does is available to you directly:
131
241
 
132
- export MOBILE_CLI_STATE=/tmp/my-target-state.json # isolate this session's device/app state
133
- bin/mcli set-device emulator-5554 # or the iOS sim UDID (xcrun simctl list devices)
134
- bin/mcli set-app com.android.chrome # or com.apple.mobilesafari, or your app's id
242
+ ```bash
243
+ bin/mcli boot qa_pixel_api35 # or an iOS sim name / UDID
244
+ bin/mcli --state /tmp/my-run.json set-device emulator-5554 --platform android
245
+ bin/mcli set-app com.android.chrome # or com.apple.mobilesafari, or your app id
135
246
  bin/mcli launch
136
- bin/mcli open-url https://staging.m.example.com # web mode only
137
- bin/mcli snapshot # page/app as tappable refs e1, e2, ...
138
- bin/mcli fill e4 my-username
247
+ bin/mcli open-url https://staging.m.example.com # web mode (alias of deep-link)
248
+ bin/mcli snapshot # a11y tree as refs e1, e2, … (refs expire after 60 s)
249
+ bin/mcli fill e4 my-username # Maestro inputText
139
250
  bin/mcli snapshot # ALWAYS re-snapshot after the keyboard appears
140
251
  bin/mcli click e7
141
- bin/mcli screenshot /tmp/evidence.png
142
- bin/mcli logs --errors --since 60
143
- bin/mcli --help # full command surface
252
+ bin/mcli fill-id login-btn "pass" # tap + type by testID, no snapshot needed
253
+ bin/mcli wait-text "Dashboard" 15
254
+ bin/mcli press HOME # BACK | ENTER | HOME | TAB | ESCAPE
255
+ bin/mcli screenshot evidence.png # no app required
256
+ bin/mcli logs --errors --since 60 # logcat / simctl log; logs-clear on Android
257
+ bin/mcli record-start run.mp4 && bin/mcli record-stop # mp4 on both platforms
258
+ bin/mcli info # resolved state + which state file is in use
259
+ bin/mcli --help
144
260
  ```
145
261
 
146
- iOS Safari web targets additionally get real JS/DOM access via the WebKit bridge:
262
+ Exit codes: `0` ok · `1` command failed · `2` usage · `3` stale/unknown ref (re-run
263
+ `snapshot`) · `4` no device or app set.
264
+
265
+ iOS Safari web targets also get real JS/DOM access through the WebKit bridge (needs
266
+ `ios-webkit-debug-proxy`, `python3` and Node >= 22.4; `WK_IOS_PORT` overrides the port):
147
267
 
148
268
  ```bash
149
- bin/wk-ios 'document.title' # eval JS in the sim's Safari
150
- bin/wk-ios 'document.querySelectorAll(".cart_item").length' # exact-DOM assertions
151
- bin/wk-ios --stop # stop the proxy after the session
269
+ bin/wk-ios 'document.title'
270
+ bin/wk-ios 'document.querySelectorAll(".cart_item").length'
271
+ bin/wk-ios --stop
152
272
  ```
153
273
 
154
274
  Notes that save time:
275
+
155
276
  - **Never `relaunch-clean` a web target** — it wipes the logged-in browser profile.
156
- - SSO/MFA logins are done **once, by hand, on the device**; the browser profile persists
157
- the session. There is no transferable `storage_state` on a real device browser.
158
- - Use `bin/wadb` for raw adb commands (logcat, `pm`, `am`) — it sets `ANDROID_HOME` for you.
159
- - Full setup guide + troubleshooting: `docs/MOBILE-SETUP.md`.
277
+ - SSO/MFA logins are done **once, by hand, on the device**. There is no transferable
278
+ `storage_state` for a real device browser.
279
+ - `bin/wadb` is raw `adb` with `ANDROID_HOME` and JDK 17 resolved for you.
280
+ - Full guide and troubleshooting: **[docs/MOBILE-SETUP.md](docs/MOBILE-SETUP.md)**.
160
281
 
161
- ### Generate or regenerate a report
282
+ ## Backend, API and infrastructure verification
162
283
 
163
- ```bash
164
- # regenerate report for an existing session
165
- /qa-explore-report output/sessions/2026-03-28-parabank
166
- ```
167
-
168
- ### Add knowledge to the base
284
+ Not every acceptance criterion is visible in a browser. `/qa-verify-backend` covers tickets
285
+ whose ACs live below the UI — tables and streams, queue consumers, Lambda triggers, IAM
286
+ policies, webhooks, IaC, and the service's own HTTP endpoints — and produces an **AC
287
+ traceability matrix** instead of a pass/fail claim.
169
288
 
170
289
  ```bash
171
- # add new heuristics, techniques, or documentation
172
- /qa-knowledge-add
173
- ```
290
+ cp data/targets/_example-backend.yml data/targets/local-my-service.yml # service behind a UI
291
+ cp data/targets/_example-api-only.yml data/targets/local-my-api.yml # HTTP API, no UI
174
292
 
175
- ### Set up a new target
176
-
177
- ```bash
178
- # configure authentication, scope, and domain for a target app
179
- /qa-target-setup
293
+ /qa-verify-backend --target local-my-service --context output/context/TICKET-123-context.md
294
+ /qa-verify-backend --target local-my-service --static-only # no cloud credentials
295
+ /qa-verify-backend --target local-my-service --no-e2e # skip the write path
296
+ /qa-verify-backend --target local-my-api --api-only --parity local-my-api-staging
180
297
  ```
181
298
 
182
- ## How to Generate Reports
183
-
184
- qualiow produces three report formats from session data.
185
-
186
- ### HTML report
187
-
188
- Self-contained dark-mode HTML with severity cards, bug details, coverage map, and recommendations. Print-friendly.
299
+ | Lane | What it does |
300
+ |------|--------------|
301
+ | **Static** | Reads the implementation branch against each AC with `git show` (never checks out) — spec drift, scope creep, failure paths, identity propagation, producer/consumer contract breaks |
302
+ | **Live** | Read-only `aws-cli` probes, one per AC, raw output saved as evidence. Never mutates; hard stop on production |
303
+ | **API** | Calls the service's own endpoints from the already-authenticated page, with a written case matrix covering everything the browser's guards prevent. Read-only, limited to the target's `api.probe_allowlist` |
304
+ | **End-to-end** | Drives the real write path in a non-production environment, then re-probes the data layer — same-tick writes, deletes, bulk saves, a second identity, DLQ depth |
305
+
306
+ Each AC gets a verdict with cited evidence — `PASS`, `PARTIAL`, `FAIL`, `BLOCKED`,
307
+ `NOT-REACHABLE` or `UNVERIFIABLE` — and the verdict **names the mode it was reached by and the
308
+ environment it holds in**. `PASS (direct request, raw response attached)` and `PASS (read the
309
+ diff)` are different claims, and a verdict backed only by a code reading is `UNVERIFIABLE`,
310
+ never `PASS`. `BLOCKED` is a first-class outcome: when credentials are missing, the probe
311
+ commands are written out ready to run rather than the verdict being inferred from source.
312
+
313
+ ### Two things it will not let you get away with
314
+
315
+ **Believing a green result from an environment that does not run your code.** Before the first
316
+ probe, the skill fingerprints every component of the request path: which build is deployed,
317
+ whether the changed path is *selected* here, and whether the commit under test is genuinely an
318
+ ancestor of what is running. One identical build can hold two implementations of the same
319
+ feature with a flag choosing between them — *same build, different behaviour ⇒ configuration,
320
+ not deploy lag*. An environment that does not run the change gets `NOT-REACHABLE`.
321
+
322
+ **Recording a client-side guard as the endpoint's behaviour.** "The search box refuses to fire
323
+ when empty" describes the browser. Every other client — mobile, partner integration, script —
324
+ sends that request, and it is frequently the one that fails. When a ticket has both a screen
325
+ and an endpoint, the same cases run at both surfaces and each finding is sorted into *both*,
326
+ *API only* (a real defect the guard is hiding), *UI only* (the client invents or masks
327
+ behaviour the service does not have) or *neither*.
328
+
329
+ The API lane needs no token handling: the request runs inside the page that is already logged
330
+ in, so it carries the same session cookie, CSRF token and interceptors as the UI — which also
331
+ means it works with SSO and MFA that no scripted login can pass. Credentials stay in the
332
+ browser profile under `.auth/` and never enter a script, a report or the repository. A service
333
+ with **no UI at all** declares `api.auth: api-key-env` and the lane runs out of browser,
334
+ reading the key from the environment variable the target names.
335
+
336
+ ### Correct shape is not a correct answer
337
+
338
+ Every derived value — a percentage, total, ratio, delta or aggregate — is recomputed from the
339
+ raw figures in the same response, with the formula taken from the specification rather than from
340
+ the code under test, and with cases chosen to stress sign, zero, scale and cardinality. Then the
341
+ report states the limitation plainly: when both sides of the check come from one payload, the
342
+ derivation is verified and the inputs are not, so the **independent oracle** that would close
343
+ the gap is named along with whether it was run.
344
+
345
+ The payload is also held next to the screen, because a correct response can still reach the user
346
+ as a wrong number — a formatter that guesses what a value is, a unit applied twice, a truncated
347
+ figure presented as a total. That defect is invisible from either surface alone, and it usually
348
+ belongs to a different change than the one under test.
349
+
350
+ ### Findings with no acceptance criterion become a spec
351
+
352
+ Most of what an API probe turns up has nothing to be filed against, so it turns into an argument
353
+ rather than a fix. `data/templates/expected-behaviour.md` converts a pile of observations into
354
+ one reviewable decision: observed against expected, grouped by cause, with the decisions the fix
355
+ forces made explicit — reject rather than clamp, an error rather than a silent zero, validation
356
+ at the layer that covers every implementation — ranked by what real users can reach today, and
357
+ closed with a plain-English reply for whoever decides to fund the work.
358
+
359
+ ### Verifying a release rather than a ticket
360
+
361
+ [`skills/qa-verify-backend/references/release-readiness.md`](skills/qa-verify-backend/references/release-readiness.md)
362
+ changes the shape of the session: the deployment table first for every ticket, so a ticket whose
363
+ backend is not in the build is marked *not testable here* rather than tested against a UI that
364
+ will render convincing nonsense; a result vocabulary that keeps *not tested* visible; a coverage
365
+ map where 🔍 *code-verified only* is marked as `UNVERIFIABLE` rather than green, with one
366
+ sentence naming the untested item that carries the most risk; carry-over defects in their own
367
+ section; and a disposition with the condition that would reverse it.
368
+
369
+ Full guide: **[docs/BACKEND-VERIFICATION.md](docs/BACKEND-VERIFICATION.md)**. Safety rules
370
+ (read-only discipline, the production hard stop, redaction):
371
+ [`skills/qa-verify-backend/references/safety-rules.md`](skills/qa-verify-backend/references/safety-rules.md).
372
+
373
+ ## Reports
374
+
375
+ `qualiow report` exports an existing session in four formats. Every export carries the
376
+ confidentiality header and runs through the same redaction list as the skills.
189
377
 
190
378
  ```bash
191
- # from TypeScript
192
- import { generateHtmlReport } from '@qualiow/exploratory-testing';
193
- const html = await generateHtmlReport('output/sessions/2026-03-28-parabank');
379
+ qualiow report -s latest -f html -o report.html
380
+ qualiow report -s 2026-09-08-1813-explore-parabank -f jira -o bugs.csv
381
+ qualiow report -s latest -f json --stdout
194
382
  ```
195
383
 
196
- ### JSON report
384
+ | Format | Output |
385
+ |--------|--------|
386
+ | `md` (default) | `session-summary.md` next to the session report — never overwrites it |
387
+ | `html` | Self-contained dark-mode page: severity cards, bug details, coverage map, recommendations. Print-friendly, CSP-restricted, `noindex` |
388
+ | `json` | Structured data for dashboards, CI and metrics tracking |
389
+ | `jira` | CSV for Jira bulk import; severity maps to priority (Critical → Blocker, High → Critical, Medium → Major, Low → Minor) |
197
390
 
198
- Structured JSON for programmatic consumption -- dashboards, CI integrations, metrics tracking.
391
+ The same formatters are exported as a library:
199
392
 
200
- ```bash
201
- # from TypeScript
202
- import { generateJsonReport } from '@qualiow/exploratory-testing';
203
- const data = await generateJsonReport('output/sessions/2026-03-28-parabank');
204
- ```
393
+ ```typescript
394
+ import {
395
+ generateHtmlReport,
396
+ generateJsonReport,
397
+ generateJiraExport,
398
+ } from 'qualiow-exploratory-testing';
205
399
 
206
- ### Jira CSV export
400
+ const session = 'output/sessions/2026-09-08-1813-explore-parabank';
207
401
 
208
- CSV importable via Jira bulk import. Maps severity to Jira priority (Critical -> Blocker, High -> Critical, Medium -> Major, Low -> Minor).
209
-
210
- ```bash
211
- # from TypeScript
212
- import { generateJiraExport } from '@qualiow/exploratory-testing';
213
- const csv = await generateJiraExport('output/sessions/2026-03-28-parabank');
402
+ const html = await generateHtmlReport(session);
403
+ const json = await generateJsonReport(session);
404
+ const csv = await generateJiraExport(session);
214
405
  ```
215
406
 
216
- ## How to Use Skills in Claude Code
407
+ ## Skills
217
408
 
218
409
  | Skill | Purpose |
219
410
  |-------|---------|
220
- | `/qa-explore` | Full exploratory testing session (45 min) |
221
- | `/qa-explore-mobile` | Exploratory session on a simulator/emulator — native apps or mobile web in the real device browser (iOS Safari / Android Chrome) |
411
+ | `/qa-explore` | Full exploratory testing session (45 min, 8 phases) |
222
412
  | `/qa-explore-quick` | Quick focused session on a single page or feature (15 min) |
223
- | `/qa-explore-report` | Generate or regenerate report from existing session |
413
+ | `/qa-explore-mobile` | Session on a simulator/emulator — native apps or web in the real device browser |
414
+ | `/qa-verify-backend` | Backend/API/infra AC verification — branch review, read-only cloud probes, API probes, traceability matrix |
415
+ | `/qa-gather` | Gather and analyze requirements from files, URLs, tickets or text into a context file |
416
+ | `/qa-explore-report` | Generate, regenerate or reformat a report from an existing session |
224
417
  | `/qa-explore-feedback` | Post-session feedback capture (false positives, missed bugs) |
225
- | `/qa-explore-cleanup` | Session cleanup and archival |
226
- | `/qa-knowledge-add` | Add new heuristics, techniques, docs to knowledge base |
227
- | `/qa-knowledge-list` | Browse the knowledge base |
228
- | `/qa-target-setup` | Configure a new target application (auth, scope, domain) |
229
- | `/qa-gather` | Gather and analyze requirements from files, URLs, or text |
418
+ | `/qa-explore-cleanup` | Session listing, archival and deletion |
419
+ | `/qa-knowledge-add` | Add heuristics, techniques, checklists or references to the knowledge base |
420
+ | `/qa-knowledge-list` | Browse and search the knowledge base |
421
+ | `/qa-target-setup` | Configure a target application (auth, scope, domain) |
230
422
 
231
- ## Project Structure
423
+ Under the plugin install these are `/qualiow:qa-explore` and so on. One sub-agent ships
424
+ alongside them: `qa-gather-agent`, the background runner behind `/qa-gather`.
425
+
426
+ ## Project structure
232
427
 
233
428
  ```
234
429
  qualiow-exploratory-testing-skills/
430
+ .claude/
431
+ skills/ # the 11 skills — CANONICAL source
432
+ agents/qa-gather-agent.md # the gather sub-agent — CANONICAL source
433
+ skills/ # generated mirror (npm + plugin) — npm run sync:plugin
434
+ agents/ # generated mirror
435
+ .claude-plugin/plugin.json # Claude Code plugin manifest
436
+ bin/
437
+ mcli, mobile-cli.mjs # Maestro/simctl/adb shim + permission-friendly wrapper
438
+ wadb, wk-ios, wkeval.mjs # adb wrapper, iOS WebKit DOM bridge
439
+ setup-mobile.sh # one-time toolchain bootstrap
440
+ doctor-mobile.sh # read-only preflight
441
+ scripts/ # repo dev tooling, not shipped:
442
+ # kb-sync.mjs, check-pack.mjs, sync-version.mjs
443
+ data/
444
+ knowledge/ # YAML knowledge base, versioned as releases v0.1.0…v0.6.0
445
+ domains/ # domain profiles (*.yml)
446
+ templates/ # bug report, session report, charter, coverage map, AC matrix
447
+ targets/ # target configs (_default, _example-*, public POC sites)
448
+ security/SECURITY-POLICY.md
449
+ docs/ # GETTING-STARTED, MOBILE-SETUP, BACKEND-VERIFICATION,
450
+ # KNOWN-ISSUES, ARCHITECTURE-DECISIONS
235
451
  src/
236
- cli/
237
- index.ts # CLI entry point (commander)
238
- commands/ # CLI command handlers
239
- formatters/
240
- html-report.ts # standalone dark-mode HTML report
241
- json-report.ts # structured JSON report
242
- jira-export.ts # Jira-importable CSV
243
- index.ts # re-exports
244
- schemas/ # Zod validation schemas
245
- types/ # TypeScript type definitions
246
- utils/
247
- metrics.ts # session metrics (JSONL append/read)
248
- parse-session.ts # parse session dirs into structured data
249
- redact.ts # credential and secret redaction
250
- validate.ts # YAML config validation
452
+ cli/ # commander entry + command handlers
453
+ formatters/ # html, json, jira, markdown summary
454
+ schemas/ # Zod schemas for targets, domains, knowledge, metrics
455
+ types/ utils/ # redaction, session parsing, paths, metrics, validation
251
456
  index.ts # public API re-exports
252
- data/
253
- knowledge/ # YAML knowledge base (heuristics, techniques)
254
- templates/ # bug report, session report, charter templates
255
- targets/ # target application configs
256
- domains/ # domain-specific testing focus
257
- security/ # security policy
258
- output/
259
- sessions/ # session outputs (reports, bugs, screenshots)
260
457
  tests/
261
- unit/ # unit tests (vitest)
262
- fixtures/ # test fixture data
263
- benchmarks/ # performance benchmarks
264
- docs/
265
- GETTING-STARTED.md # detailed setup and usage guide
266
- ARCHITECTURE-DECISIONS.md # architecture decision records
458
+ unit/ fixtures/ # vitest
459
+ output/
460
+ sessions/ bugs/ context/
267
461
  ```
268
462
 
269
- ## Tech Stack
463
+ In a project you ran `qualiow init` in:
464
+
465
+ ```
466
+ my-project/
467
+ .claude/skills/qa-* # the 11 skills
468
+ .claude/agents/qa-gather-agent.md
469
+ data/ # knowledge, domains, templates, security, targets
470
+ qa/
471
+ target.yml # project-local target (from /qa-target-setup)
472
+ .env # credentials — gitignored
473
+ .env.example
474
+ bin/ # mobile driver + setup/doctor scripts
475
+ output/sessions|bugs|context
476
+ .auth/ # storage states and browser profiles — gitignored
477
+ ```
478
+
479
+ ## Tech stack
270
480
 
271
481
  | Technology | Purpose |
272
482
  |------------|---------|
273
- | TypeScript | type-safe source code |
274
- | Node.js >= 18 | runtime |
275
- | Playwright CLI | browser control for AI agents |
276
483
  | Claude Code | AI reasoning and session orchestration |
277
- | Handlebars | HTML report templating |
484
+ | Playwright CLI (`@playwright/cli`) | browser control for the web path |
485
+ | Maestro + `xcrun simctl` / `adb` | device control for the mobile path |
486
+ | TypeScript / Node.js >= 22.4 | CLI, formatters, schemas |
278
487
  | Zod | runtime schema validation |
279
488
  | Commander | CLI argument parsing |
280
- | Vitest | unit and integration testing |
281
- | tsup | TypeScript bundling |
489
+ | Handlebars | HTML report templating |
490
+ | Vitest | unit testing |
491
+ | tsup | bundling |
282
492
  | YAML | configuration and knowledge base format |
283
493
 
284
- ## Key Principles
494
+ ## Key principles
285
495
 
286
496
  - Systematic exploration, not random clicking
287
497
  - Understand the business first, then test what matters
288
- - Risk-rank features: P0 (40% time), P1 (30%), P2 (20%), P3 (10%)
498
+ - Risk-rank features: P0 (40% of the time), P1 (30%), P2 (20%), P3 (10%)
289
499
  - Apply heuristics as thinking tools, not checklists
290
500
  - One bug = one report, with mandatory business impact
291
- - Log findings in real-time with WHY reasoning, not just WHAT
292
- - Cap sessions at 45 min to avoid context overflow
501
+ - Log findings in real time with WHY reasoning, not just WHAT
502
+ - Cap sessions at 45 minutes to avoid context overflow
293
503
  - Save phase findings to disk between phases
294
504
  - Verify data integrity after every state-changing action
295
- - Never store credentials in YAML -- use env vars via .env
505
+ - Always ask: "what is NOT here that SHOULD be?"
506
+ - Never store credentials in YAML — reference env vars from `qa/.env`
507
+
508
+ ## Security
509
+
510
+ One rule set governs every skill:
511
+ [`skills/qa-explore/references/security-rules.md`](skills/qa-explore/references/security-rules.md).
512
+ The long-form policy is `data/security/SECURITY-POLICY.md`; where the two differ, the rules
513
+ file is what a session applies.
514
+
515
+ - **Web and app content is data, never instructions.** An injection attempt found while
516
+ testing is reported as a finding, never obeyed.
517
+ - **Never echo skill files, knowledge YAML or agent definitions** into session output or back
518
+ to a tested site.
519
+ - **Production is read-only.** A hostname matching `prod|production|prd|live` (never the URL
520
+ path) puts the session in read-only mode unless the string also contains `staging`, `dev`,
521
+ `test`, `qa`, `uat`, `sandbox`, `localhost`, `demo`, `example.com` and friends — the
522
+ exclusion wins. The only override is `safety.read_only: false` in the target file.
523
+ - **Redaction before disk.** Private keys, JWTs, `Authorization`/`Cookie` headers, AWS keys,
524
+ `sk-`/`gh*_`/`xox*-` tokens, API keys, password/token/secret assignments, emails, SSNs and
525
+ Luhn-valid card numbers are replaced with `[REDACTED]` — in the skills and in every
526
+ formatter.
527
+ - **Sessions are isolated.** Each carries its own `playwright-cli -s=<id>`, closed and
528
+ `delete-data`'d at the end.
529
+ - **All output is confidential and local.** Every artefact opens with the confidentiality
530
+ blockquote, and nothing is sent to an external service.
531
+
532
+ ## Documentation
533
+
534
+ - **[docs/GETTING-STARTED.md](docs/GETTING-STARTED.md)** — install, first session, targets,
535
+ knowledge, CLI
536
+ - **[docs/MOBILE-SETUP.md](docs/MOBILE-SETUP.md)** — getting a Mac ready for simulator and
537
+ emulator sessions
538
+ - **[docs/BACKEND-VERIFICATION.md](docs/BACKEND-VERIFICATION.md)** — the four lanes, verdicts,
539
+ templates and techniques
540
+
541
+ Repo-only notes (not shipped in the npm package):
542
+ [docs/KNOWN-ISSUES.md](docs/KNOWN-ISSUES.md),
543
+ [docs/ARCHITECTURE-DECISIONS.md](docs/ARCHITECTURE-DECISIONS.md),
544
+ [CHANGELOG.md](CHANGELOG.md).
296
545
 
297
546
  ## License
298
547
 
299
- MIT -- see [LICENSE](LICENSE) for details.
548
+ MIT — see [LICENSE](LICENSE). Author: William Oliveira.