qualiow-exploratory-testing 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +177 -0
- package/bin/mcli +16 -0
- package/bin/mobile-cli.mjs +573 -0
- package/bin/wadb +6 -0
- package/bin/wk-ios +58 -0
- package/bin/wkeval.mjs +44 -0
- package/data/domains/_default.md +181 -0
- package/data/domains/ecommerce.md +192 -0
- package/data/domains/fintech.md +200 -0
- package/data/domains/identity.md +88 -0
- package/data/domains/identity.yml +138 -0
- package/data/domains/marketing.md +194 -0
- package/data/domains/saas.md +197 -0
- package/data/knowledge/changelog.yml +86 -0
- package/data/knowledge/learned-patterns.md +172 -1
- package/data/knowledge/manifest.yml +210 -4
- package/data/knowledge/releases/v0.2.0/entries/technique-contract-narrowing.yml +150 -0
- package/data/knowledge/releases/v0.2.0/entries/technique-test-suite-audit.yml +116 -0
- package/data/knowledge/releases/v0.2.0/release.yml +20 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-functional-diff-analysis.yml +119 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-llm-output-verification.yml +102 -0
- package/data/knowledge/releases/v0.3.0/entries/technique-verification-mode-selection.yml +153 -0
- package/data/knowledge/releases/v0.3.0/release.yml +25 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-authenticated-api-probing.yml +137 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-environment-fingerprinting.yml +113 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-silent-failure-audit.yml +86 -0
- package/data/knowledge/releases/v0.4.0/entries/technique-ui-api-differential.yml +97 -0
- package/data/knowledge/releases/v0.4.0/release.yml +28 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-config-surface-verification.yml +77 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-derived-value-verification.yml +81 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-expected-behaviour-specification.yml +97 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-presentation-integrity.yml +78 -0
- package/data/knowledge/releases/v0.5.0/entries/technique-release-readiness-verification.yml +110 -0
- package/data/knowledge/releases/v0.5.0/release.yml +33 -0
- package/data/targets/_example-backend.yml +123 -0
- package/data/targets/_example-mobile-emulation.yml +52 -0
- package/data/targets/_example-native-mobile.yml +61 -0
- package/data/targets/_example-sim-android-chrome.yml +81 -0
- package/data/targets/_example-sim-ios-safari.yml +80 -0
- package/data/templates/api-probe-matrix.md +99 -0
- package/data/templates/coverage-map.md +14 -0
- package/data/templates/expected-behaviour.md +93 -0
- package/dist/cli/index.js +137 -5
- package/dist/cli/index.js.map +1 -1
- package/dist/index.d.ts +350 -527
- package/dist/index.js +144 -5
- package/dist/index.js.map +1 -1
- package/docs/BACKEND-VERIFICATION.md +277 -0
- package/docs/MOBILE-SETUP.md +92 -0
- package/package.json +26 -12
- package/scripts/doctor-mobile.sh +143 -0
- package/scripts/setup-mobile.sh +213 -0
- package/skills/qa-explore/phases/00-setup.md +8 -1
- package/skills/qa-explore/phases/01-auth.md +4 -0
- package/skills/qa-explore-mobile/SKILL.md +146 -0
- package/skills/qa-explore-mobile/phases/00-setup.md +203 -0
- package/skills/qa-explore-mobile/phases/01-auth.md +127 -0
- package/skills/qa-explore-mobile/phases/02-charter.md +70 -0
- package/skills/qa-explore-mobile/phases/03-discovery.md +79 -0
- package/skills/qa-explore-mobile/phases/04-journeys.md +49 -0
- package/skills/qa-explore-mobile/phases/05-features.md +72 -0
- package/skills/qa-explore-mobile/phases/06-edge-cases.md +157 -0
- package/skills/qa-explore-mobile/phases/07-reporting.md +123 -0
- package/skills/qa-explore-mobile/references/mobile-edges.md +117 -0
- package/skills/qa-explore-quick/SKILL.md +6 -0
- package/skills/qa-verify-backend/SKILL.md +163 -0
- package/skills/qa-verify-backend/phases/00-setup.md +97 -0
- package/skills/qa-verify-backend/phases/01-ac-decomposition.md +58 -0
- package/skills/qa-verify-backend/phases/02-static-review.md +71 -0
- package/skills/qa-verify-backend/phases/03-live-verification.md +62 -0
- package/skills/qa-verify-backend/phases/03b-api-verification.md +157 -0
- package/skills/qa-verify-backend/phases/04-e2e-trigger.md +66 -0
- package/skills/qa-verify-backend/phases/05-reporting.md +135 -0
- package/skills/qa-verify-backend/references/api-probes.md +141 -0
- package/skills/qa-verify-backend/references/aws-readonly-probes.md +156 -0
- package/skills/qa-verify-backend/references/environment-fingerprinting.md +104 -0
- package/skills/qa-verify-backend/references/payload-verification.md +112 -0
- package/skills/qa-verify-backend/references/release-readiness.md +97 -0
- package/skills/qa-verify-backend/references/safety-rules.md +90 -0
package/README.md
CHANGED
|
@@ -74,6 +74,90 @@ npm test
|
|
|
74
74
|
/qa-explore-quick https://app.example.com/checkout
|
|
75
75
|
```
|
|
76
76
|
|
|
77
|
+
### Mobile session (simulator/emulator)
|
|
78
|
+
|
|
79
|
+
Runs a full exploratory session on an iOS Simulator or Android Emulator — either a
|
|
80
|
+
**native app** (the installed app is the system under test) or a **mobile web app in the
|
|
81
|
+
real device browser** (iOS Simulator Safari / Android Emulator Chrome — the genuine
|
|
82
|
+
engines, which Playwright cannot drive). The mode comes from the target config.
|
|
83
|
+
|
|
84
|
+
#### 1. One-time machine setup
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
# installs everything scriptable: Maestro, JDK 17, Android SDK + Google-Play AVD
|
|
88
|
+
# (qa_pixel_api35), ios-webkit-debug-proxy, and the qa-iphone simulator device
|
|
89
|
+
scripts/setup-mobile.sh # add --android or --ios to scope; --yes for non-interactive
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Two iOS steps are genuinely manual (the script prints them if needed):
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
# a) install Xcode from the App Store, then:
|
|
96
|
+
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
|
|
97
|
+
sudo xcodebuild -license accept
|
|
98
|
+
# b) if no iOS runtime is installed:
|
|
99
|
+
xcodebuild -downloadPlatform iOS
|
|
100
|
+
# then re-run scripts/setup-mobile.sh --ios (creates the qa-iphone simulator)
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
#### 2. Verify readiness (read-only, run before every first session on a machine)
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
scripts/doctor-mobile.sh # or --ios / --android; fix each ✗ until READY
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
#### 3. Create a target and run
|
|
110
|
+
|
|
111
|
+
Copy a template from `data/targets/`, rename it, and point it at your app:
|
|
112
|
+
|
|
113
|
+
| Template | What it tests | Driver |
|
|
114
|
+
|----------|---------------|--------|
|
|
115
|
+
| `_example-native-mobile.yml` | An installed iOS/Android app (native mode) | mobile-cli (Maestro) |
|
|
116
|
+
| `_example-sim-ios-safari.yml` | A mobile web app in REAL iOS Simulator Safari | mobile-cli (Maestro) |
|
|
117
|
+
| `_example-sim-android-chrome.yml` | A mobile web app in REAL Android Emulator Chrome | mobile-cli (Maestro) |
|
|
118
|
+
| `_example-mobile-emulation.yml` | A mobile web app via Playwright device emulation (no simulator) | `/qa-explore` (playwright-cli) |
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
/qa-explore-mobile --target my-mobile-target # real sim/emulator (native or web)
|
|
122
|
+
/qa-explore --target my-emulation-target # Playwright mobile emulation
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
#### 4. Driving the device manually (what the skill does under the hood)
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
# Android: boot the standard AVD # iOS: boot the standard simulator
|
|
129
|
+
~/Library/Android/sdk/emulator/emulator \
|
|
130
|
+
-avd qa_pixel_api35 -no-snapshot & xcrun simctl boot qa-iphone && open -a Simulator
|
|
131
|
+
|
|
132
|
+
export MOBILE_CLI_STATE=/tmp/my-target-state.json # isolate this session's device/app state
|
|
133
|
+
bin/mcli set-device emulator-5554 # or the iOS sim UDID (xcrun simctl list devices)
|
|
134
|
+
bin/mcli set-app com.android.chrome # or com.apple.mobilesafari, or your app's id
|
|
135
|
+
bin/mcli launch
|
|
136
|
+
bin/mcli open-url https://staging.m.example.com # web mode only
|
|
137
|
+
bin/mcli snapshot # page/app as tappable refs e1, e2, ...
|
|
138
|
+
bin/mcli fill e4 my-username
|
|
139
|
+
bin/mcli snapshot # ALWAYS re-snapshot after the keyboard appears
|
|
140
|
+
bin/mcli click e7
|
|
141
|
+
bin/mcli screenshot /tmp/evidence.png
|
|
142
|
+
bin/mcli logs --errors --since 60
|
|
143
|
+
bin/mcli --help # full command surface
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
iOS Safari web targets additionally get real JS/DOM access via the WebKit bridge:
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
bin/wk-ios 'document.title' # eval JS in the sim's Safari
|
|
150
|
+
bin/wk-ios 'document.querySelectorAll(".cart_item").length' # exact-DOM assertions
|
|
151
|
+
bin/wk-ios --stop # stop the proxy after the session
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Notes that save time:
|
|
155
|
+
- **Never `relaunch-clean` a web target** — it wipes the logged-in browser profile.
|
|
156
|
+
- SSO/MFA logins are done **once, by hand, on the device**; the browser profile persists
|
|
157
|
+
the session. There is no transferable `storage_state` on a real device browser.
|
|
158
|
+
- Use `bin/wadb` for raw adb commands (logcat, `pm`, `am`) — it sets `ANDROID_HOME` for you.
|
|
159
|
+
- Full setup guide + troubleshooting: `docs/MOBILE-SETUP.md`.
|
|
160
|
+
|
|
77
161
|
### Generate or regenerate a report
|
|
78
162
|
|
|
79
163
|
```bash
|
|
@@ -95,6 +179,96 @@ npm test
|
|
|
95
179
|
/qa-target-setup
|
|
96
180
|
```
|
|
97
181
|
|
|
182
|
+
## How to Verify Backend Acceptance Criteria
|
|
183
|
+
|
|
184
|
+
Not every acceptance criterion is visible in a browser. `/qa-verify-backend` covers tickets
|
|
185
|
+
whose ACs live below the UI — tables and streams, queue consumers, Lambda triggers, IAM
|
|
186
|
+
policies, webhooks, IaC, and the service's own HTTP endpoints — and produces an **AC
|
|
187
|
+
traceability matrix** instead of a pass/fail claim.
|
|
188
|
+
|
|
189
|
+
```bash
|
|
190
|
+
# Copy the template, point it at your service, then verify a ticket branch
|
|
191
|
+
cp data/targets/_example-backend.yml data/targets/local-my-service.yml
|
|
192
|
+
|
|
193
|
+
/qa-verify-backend --target local-my-service --context output/context/TICKET-123-context.md
|
|
194
|
+
/qa-verify-backend --target local-my-service --static-only # no cloud credentials
|
|
195
|
+
/qa-verify-backend --target local-my-service --api-only --parity local-my-service-staging
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
It runs four lanes:
|
|
199
|
+
|
|
200
|
+
| Lane | What it does |
|
|
201
|
+
|------|--------------|
|
|
202
|
+
| **Static** | Reads the implementation branch against each AC with `git show` (never checks out) — spec drift, scope creep, failure paths, identity propagation, producer/consumer contract breaks |
|
|
203
|
+
| **Live** | Read-only `aws-cli` probes, one per AC, raw output saved as evidence. Never mutates; hard stop on production |
|
|
204
|
+
| **API** | Calls the service's own endpoints from the already-authenticated page, with a written case matrix covering everything the browser's guards prevent. Read-only, limited to the target's `api.probe_allowlist` |
|
|
205
|
+
| **End-to-end** | Drives the real write path in a non-production environment, then re-probes the data layer — including same-tick writes, deletes, bulk saves, a second identity, and DLQ depth |
|
|
206
|
+
|
|
207
|
+
Each AC gets a verdict with cited evidence — `PASS`, `PARTIAL`, `FAIL`, `BLOCKED`,
|
|
208
|
+
`NOT-REACHABLE` or `UNVERIFIABLE` — and the verdict **names the mode it was reached by and the
|
|
209
|
+
environment it holds in**. `PASS (direct request, raw response attached)` and `PASS (read the
|
|
210
|
+
diff)` are different claims, and a verdict backed only by a code reading is `UNVERIFIABLE`,
|
|
211
|
+
never `PASS`. `BLOCKED` is a first-class outcome: when credentials are missing, the probe
|
|
212
|
+
commands are written out ready to run rather than the verdict being inferred from source.
|
|
213
|
+
|
|
214
|
+
### Two things it will not let you get away with
|
|
215
|
+
|
|
216
|
+
**Believing a green result from an environment that does not run your code.** Before the first
|
|
217
|
+
probe, the skill fingerprints every component of the request path: which build is deployed,
|
|
218
|
+
whether the changed path is *selected* here, and whether the commit under test is genuinely an
|
|
219
|
+
ancestor of what is running. One identical build can hold two implementations of the same
|
|
220
|
+
feature with a flag choosing between them — *same build, different behaviour ⇒ configuration,
|
|
221
|
+
not deploy lag*. An environment that does not run the change gets `NOT-REACHABLE`.
|
|
222
|
+
|
|
223
|
+
**Recording a client-side guard as the endpoint's behaviour.** "The search box refuses to fire
|
|
224
|
+
when empty" describes the browser. Every other client — mobile, partner integration, script —
|
|
225
|
+
sends that request, and it is frequently the one that fails. When a ticket has both a screen
|
|
226
|
+
and an endpoint, the same cases run at both surfaces and each finding is sorted into *both*,
|
|
227
|
+
*API only* (a real defect the guard is hiding), *UI only* (the client invents or masks
|
|
228
|
+
behaviour the service does not have) or *neither*.
|
|
229
|
+
|
|
230
|
+
The API lane needs no token handling: the request runs inside the page that is already logged
|
|
231
|
+
in, so it carries the same session cookie, CSRF token and interceptors as the UI — which also
|
|
232
|
+
means it works with SSO and MFA that no scripted login can pass. Credentials stay in the
|
|
233
|
+
browser profile under `.auth/` and never enter a script, a report or the repository.
|
|
234
|
+
|
|
235
|
+
### Correct shape is not a correct answer
|
|
236
|
+
|
|
237
|
+
Every derived value — a percentage, total, ratio, delta or aggregate — is recomputed from the
|
|
238
|
+
raw figures in the same response, with the formula taken from the specification rather than from
|
|
239
|
+
the code under test, and with cases chosen to stress sign, zero, scale and cardinality. Then the
|
|
240
|
+
report states the limitation plainly: when both sides of the check come from one payload, the
|
|
241
|
+
derivation is verified and the inputs are not, so the **independent oracle** that would close
|
|
242
|
+
the gap is named along with whether it was run.
|
|
243
|
+
|
|
244
|
+
The payload is also held next to the screen, because a correct response can still reach the user
|
|
245
|
+
as a wrong number — a formatter that guesses what a value is, a unit applied twice, a truncated
|
|
246
|
+
figure presented as a total. That defect is invisible from either surface alone, and it usually
|
|
247
|
+
belongs to a different change than the one under test.
|
|
248
|
+
|
|
249
|
+
### Findings with no acceptance criterion become a spec
|
|
250
|
+
|
|
251
|
+
Most of what an API probe turns up has nothing to be filed against, so it turns into an argument
|
|
252
|
+
rather than a fix. `data/templates/expected-behaviour.md` converts a pile of observations into
|
|
253
|
+
one reviewable decision: observed against expected, grouped by cause, with the decisions the fix
|
|
254
|
+
forces made explicit — reject rather than clamp, an error rather than a silent zero, validation
|
|
255
|
+
at the layer that covers every implementation — ranked by what real users can reach today, and
|
|
256
|
+
closed with a plain-English reply for whoever decides to fund the work.
|
|
257
|
+
|
|
258
|
+
### Verifying a release rather than a ticket
|
|
259
|
+
|
|
260
|
+
`.claude/skills/qa-verify-backend/references/release-readiness.md` changes the shape of the
|
|
261
|
+
session: the deployment table first
|
|
262
|
+
for every ticket, so a ticket whose backend is not in the build is marked *not testable here*
|
|
263
|
+
rather than tested against a UI that will render convincing nonsense; a result vocabulary that
|
|
264
|
+
keeps *not tested* visible; a coverage map where 🔍 *code-verified only* is marked as
|
|
265
|
+
`UNVERIFIABLE` rather than green, with one sentence naming the untested item that carries the
|
|
266
|
+
most risk; carry-over defects in their own section; and a disposition with the condition that
|
|
267
|
+
would reverse it.
|
|
268
|
+
|
|
269
|
+
Full guide: **`docs/BACKEND-VERIFICATION.md`**. Safety rules (read-only discipline, the
|
|
270
|
+
production hard stop, redaction): `.claude/skills/qa-verify-backend/references/safety-rules.md`.
|
|
271
|
+
|
|
98
272
|
## How to Generate Reports
|
|
99
273
|
|
|
100
274
|
qualiow produces three report formats from session data.
|
|
@@ -134,7 +308,9 @@ const csv = await generateJiraExport('output/sessions/2026-03-28-parabank');
|
|
|
134
308
|
| Skill | Purpose |
|
|
135
309
|
|-------|---------|
|
|
136
310
|
| `/qa-explore` | Full exploratory testing session (45 min) |
|
|
311
|
+
| `/qa-explore-mobile` | Exploratory session on a simulator/emulator — native apps or mobile web in the real device browser (iOS Safari / Android Chrome) |
|
|
137
312
|
| `/qa-explore-quick` | Quick focused session on a single page or feature (15 min) |
|
|
313
|
+
| `/qa-verify-backend` | Verify backend/API/infra acceptance criteria with no UI surface — branch review, read-only cloud probes, direct API probes, AC traceability matrix |
|
|
138
314
|
| `/qa-explore-report` | Generate or regenerate report from existing session |
|
|
139
315
|
| `/qa-explore-feedback` | Post-session feedback capture (false positives, missed bugs) |
|
|
140
316
|
| `/qa-explore-cleanup` | Session cleanup and archival |
|
|
@@ -178,6 +354,7 @@ qualiow-exploratory-testing-skills/
|
|
|
178
354
|
benchmarks/ # performance benchmarks
|
|
179
355
|
docs/
|
|
180
356
|
GETTING-STARTED.md # detailed setup and usage guide
|
|
357
|
+
BACKEND-VERIFICATION.md # verifying ACs with no UI surface
|
|
181
358
|
ARCHITECTURE-DECISIONS.md # architecture decision records
|
|
182
359
|
```
|
|
183
360
|
|
package/bin/mcli
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
#!/bin/bash
|
|
2
|
+
# mcli — env-setting wrapper around mobile-cli.mjs so subagents can call a single
|
|
3
|
+
# bash command without needing to prefix every call with export JAVA_HOME=... etc.
|
|
4
|
+
# (Compound `export ...; cmd ...` calls hit the Claude Code permission system
|
|
5
|
+
# which matches on the first token of the line.)
|
|
6
|
+
# Resolve JDK 17 portably: honor an existing JAVA_HOME, else ask macOS' java_home,
|
|
7
|
+
# else fall back to the Homebrew prefixes (arm64 then Intel). Avoids hardcoding one path.
|
|
8
|
+
if [ -z "${JAVA_HOME:-}" ]; then
|
|
9
|
+
JAVA_HOME="$(/usr/libexec/java_home -v 17 2>/dev/null || true)"
|
|
10
|
+
[ -z "$JAVA_HOME" ] && [ -d /opt/homebrew/opt/openjdk@17 ] && JAVA_HOME=/opt/homebrew/opt/openjdk@17
|
|
11
|
+
[ -z "$JAVA_HOME" ] && [ -d /usr/local/opt/openjdk@17 ] && JAVA_HOME=/usr/local/opt/openjdk@17
|
|
12
|
+
fi
|
|
13
|
+
export JAVA_HOME
|
|
14
|
+
export ANDROID_HOME="${ANDROID_HOME:-$HOME/Library/Android/sdk}"
|
|
15
|
+
export PATH="$JAVA_HOME/bin:$ANDROID_HOME/platform-tools:$ANDROID_HOME/emulator:$HOME/.maestro/bin:$PATH"
|
|
16
|
+
exec node "$(dirname "$0")/mobile-cli.mjs" "$@"
|